Infrastructure Team Meeting - October 10, 2023

[dduportal (Damien Duportal)]

2023 10 10 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: 2.427 bits are out (war, packages, container image, changelog)
2. Curl 8.4.0 the 11th

Upcoming Calendar

• Next Weekly: 2.428, 2023-10-17
• Next LTS: 2.414.3, 2023-10-18
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • DevOps World Santa Clara Oct. 18-19 2023
  • FOSDEM Feb. 2024 (3-4 Feb.)

Notes

• Done:

  • Migrate to GitHub documentation on sonar plugin
  • Update machines to maven 3.9.5
  • Can not retrieve my Jenkins account
  • [Packer image 1.27.1] Windows VM failing to allocate
  • not able to log in
  • Access to GitHub Packages in the plugin
  • [publick8s] Ensure high availability of replicated services
    • We can plan Kube 1.26 upgrade now!
  • Remove Oracle Cloud project and tools

• Work In Progress:

  • Plugin Site API is not building since february
    • Build is fixed and container image are deployed to DockerHub
    • Next step: deliver new version to production. Watch it!
    • Btw: this service (API backend only!) should be sunset soon
  • plugin site backend OOM killed
    • Blocked by the above
  • Unable to Sign Up
    • Waiting for user answer, about chaning their email
  • JIRA email status from sendgrid
    • Did not have time to spend on it
    • Back to backlog
  • Packer: GOSS version tracking and moving sanity check to goss
    • Working on moving sanity checks from provisioning scripts to GOSS test harness
    • ASDF is a challenge (user, shell and PATH issues). WiP. on asdf and npm
  • Migrate Terraform states from AWS S3 to Azure buckets
    • Code ready, need to apply creation of new buckets/azure
  • speed up the Docker image library to create/push tags at the same time for both GH and Docker (instead of running additional build)
    • WiP: selected the docker-confluence-data (wiki.jenkins.io web service) as “test” and PR opened to remove tag discovery on it on infra.ci as a requirement
    • Next step: write down “unit tests” on the library to describe the new expectation of the new pipeline
  • Remove account request field from Jira login page
    • Back to backlog (after pinging other JIRA admins)
  • Upgrade to Kubernetes 1.26
    • First step done: kubectl is now 1.26.9 on our system (client-side)
    • HA is now enforced on all replicated services
    • Next step: reading changelog and preparing DigitalOcean upgrade (need a timeline!)
      • Let’s plan for after the LTS
      • We can start chaneglog and prepare DO PR though
  • Proposal for application in publick8s to migrate to arm64
    • WiP on rating.jenkins.io.
      • Had to fix the build. Done: need to deploy the new version before changing to arm64.
      • Found an issue on rating.issue.io: IPv6 client are failing. Will need a fix + a production operation.
        • After LTS then (next week)
    • Proposal for next one: private Nginx ingress controllers
  • Matomo github/docker repos
    • G.A. v4 is a problem and due to we can’t upgrade, no stats since end of July => prioritized
    • WiP: adding a matomo database with is own user
    • WiP: helm chart, to be tested without ingress (public access) at first
  • Ensure removed jenkins.io pages aren’t accessible and indexed anymore
    • We need help from the docs. team to check the pages to be deleted
    • Back to backlog, until the checks are done
  • Planning for supported JDK versions in Jenkins Infrastructure
    • JDK19 removal:
      • No plugin found with JDK19
      • From packer image (1.30.0). Wip: to be deployed on ci.jenkins.io
      • Next step: jenkins tool and docker-inbounds-agents (windows container images)
    • JDK21 available (at least a first subset of installers)
      • Let’s check the platforms (x86, amd64, but s390x? And Windows?)
    • Write infrastructure JDK support timeline
      • What about “next/edge” version such as JDK22?
      • JDK LTS support end of life?
      • Need discussion and writing (mention cost of a lot of tests/agents)
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • WiP: trusted agent to upload UC generated data to R2 (Cloudflare) and Azure fileshare
      • Tools installed
      • Credentials generated and deployed
        • AWS S3 on the jenkins user as the non-default profile
        • AzCopy uses SAS token, provided as a Jenkins credential
      • jenkins-infra/update_center2 now has a PR for this, but failing: tests in progress in a copy of the job (works locally so almost there)
      • Goal: test the new update_center2 website (with mirrors and Cloudflare)
      • Note: performance! <3min (or not all 3 min)
    • WiP (helm chart)
      • The new helm chart (featuring umbrella PVC and ingress) is deployed but breaks the mirrorbits
        • Hack: Disable httpd as an ingress backend
        • Related to a non obvious path with regex and nginx locations
      • WiP on templatizing the mirorrbits config. No priority on this now.

• ToDo (next milestone) (infra-team-sync-2023-10-17 Milestone · GitHub)

  • Plugins Backend is unhealthy - Error 54113 · Issue #3779 · jenkins-infra/helpdesk · GitHub