Infrastructure Team Meeting - October 10, 2023

dduportal · October 11, 2023, 12:11pm

Attendees

@dduportal (Damien Duportal)
@hlemeur (Hervé Le Meur)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@poddingue (Bruno Verachten)
@kmartens27 (Kevin Martens)

Announcements

Weekly: 2.427 bits are out (war, packages, container image, changelog)
Curl 8.4.0 the 11th

Upcoming Calendar

Next Weekly: 2.428, 2023-10-17
Next LTS: 2.414.3, 2023-10-18
Next Security Release as per jenkinsci-advisories: N.A.
Next major event:
- DevOps World Santa Clara Oct. 18-19 2023
- FOSDEM Feb. 2024 (3-4 Feb.)

Notes

Done:
Work In Progress:
- Plugin Site API is not building since february
  - Build is fixed and container image are deployed to DockerHub
  - Next step: deliver new version to production. Watch it!
  - Btw: this service (API backend only!) should be sunset soon
- plugin site backend OOM killed
  - Blocked by the above
- Unable to Sign Up
  - Waiting for user answer, about chaning their email
- JIRA email status from sendgrid
  - Did not have time to spend on it
  - Back to backlog
- Packer: GOSS version tracking and moving sanity check to goss
  - Working on moving sanity checks from provisioning scripts to GOSS test harness
  - ASDF is a challenge (user, shell and PATH issues). WiP. on asdf and npm
- Migrate Terraform states from AWS S3 to Azure buckets
  - Code ready, need to apply creation of new buckets/azure
- speed up the Docker image library to create/push tags at the same time for both GH and Docker (instead of running additional build)
  - WiP: selected the docker-confluence-data (wiki.jenkins.io web service) as “test” and PR opened to remove tag discovery on it on infra.ci as a requirement
  - Next step: write down “unit tests” on the library to describe the new expectation of the new pipeline
- Remove account request field from Jira login page
  - Back to backlog (after pinging other JIRA admins)
- Upgrade to Kubernetes 1.26
  - First step done: kubectl is now 1.26.9 on our system (client-side)
  - HA is now enforced on all replicated services
  - Next step: reading changelog and preparing DigitalOcean upgrade (need a timeline!)
    - Let’s plan for after the LTS
    - We can start chaneglog and prepare DO PR though
- Proposal for application in publick8s to migrate to arm64
  - WiP on rating.jenkins.io.
    - Had to fix the build. Done: need to deploy the new version before changing to arm64.
    - Found an issue on rating.issue.io: IPv6 client are failing. Will need a fix + a production operation.
      - After LTS then (next week)
  - Proposal for next one: private Nginx ingress controllers
- Matomo github/docker repos
  - G.A. v4 is a problem and due to we can’t upgrade, no stats since end of July => prioritized
  - WiP: adding a matomo database with is own user
  - WiP: helm chart, to be tested without ingress (public access) at first
- Ensure removed jenkins.io pages aren’t accessible and indexed anymore
  - We need help from the docs. team to check the pages to be deleted
  - Back to backlog, until the checks are done
- Planning for supported JDK versions in Jenkins Infrastructure
  - JDK19 removal:
    - No plugin found with JDK19
    - From packer image (1.30.0). Wip: to be deployed on ci.jenkins.io
    - Next step: jenkins tool and docker-inbounds-agents (windows container images)
  - JDK21 available (at least a first subset of installers)
    - Let’s check the platforms (x86, amd64, but s390x? And Windows?)
  - Write infrastructure JDK support timeline
    - What about “next/edge” version such as JDK22?
    - JDK LTS support end of life?
    - Need discussion and writing (mention cost of a lot of tests/agents)
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  - WiP: trusted agent to upload UC generated data to R2 (Cloudflare) and Azure fileshare
    - Tools installed
    - Credentials generated and deployed
      - AWS S3 on the jenkins user as the non-default profile
      - AzCopy uses SAS token, provided as a Jenkins credential
    - jenkins-infra/update_center2 now has a PR for this, but failing: tests in progress in a copy of the job (works locally so almost there)
    - Goal: test the new update_center2 website (with mirrors and Cloudflare)
    - Note: performance! <3min (or not all 3 min)
  - WiP (helm chart)
    - The new helm chart (featuring umbrella PVC and ingress) is deployed but breaks the mirrorbits
      - Hack: Disable httpd as an ingress backend
      - Related to a non obvious path with regex and nginx locations
    - WiP on templatizing the mirorrbits config. No priority on this now.
ToDo (next milestone) (infra-team-sync-2023-10-17 Milestone · GitHub)
- Plugins Backend is unhealthy - Error 54113 · Issue #3779 · jenkins-infra/helpdesk · GitHub