Infrastructure Team Meeting - October 24, 2023

[dduportal (Damien Duportal)]

- YouTube

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly:
   • 2.429 WAR + packaged
   • Docker image: delayed a few hours to add back the missing container tags (-jdk17)
   • Changelog merged
2. “Missing -jdk17 container tags”

• LTS changelog for 2.414.3 updated to warn users
• Fix for today’s weekly (see above)
• We plan to backport missing tags to 2.414.3 using binary copy with skopeo to avoid rebuild (and accidental overrides)

3. Remoting versions 3176 and 3180 had a bug when using “direct connection” (no websockets, no TCP discovery)

• ci.jenkins.io was impacted but no user facing problem
• version 3181 of remoting fixed the issue
• rolling it out to ci.jenkins.io

4. Jenkins Election

• Nominations ends soon (27 Oct.)
• Voter registration!
• See Jenkins Board and Officer Elections 2023 - Nominations Open for details

5. Team availabilities

• 1st and 11 of November: no French Jenkins SRE!
• Expect a bit less availability from the team

Upcoming Calendar

• Next Weekly: 2.430 next week
• Next LTS: November 15, RC the 1st of November
• Next Security Release as per jenkinsci-advisories: https://groups.google.com/g/jenkinsci-advisories/c/1wx_QuqkKSE (plugins only)
• Next major event:
  • FOSDEM at Brussels - Feb. 2024 (3-4 Feb.)

Notes

• Done:

  • Lost write access to credentials-plugin
  • Need Github permissions for Developers
  • Create a coverage-plugin component in the JENKINS Jira project
  • Archive Jira component extra-tool-installers-plugin
  • Wrong plugin redirect
  • Trying to create a new jenkins community account
  • Migrate Terraform states from AWS S3 to Azure buckets
  • plugin-site build commonly fails on infra.ci when accessing https://plugins.jenkins.io resulting in a 502

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Knowledge sharing done (Herve → Stephane)
    • WIP: update-center2 script executed every 3 minute
      • Parallelization applied on the copy of JSON to different targets (rsync to VM, AWS S3 sync to R2, Azcopy to Azure filestorage). Timing looks really good
      • symlinks issue fixed by using a local rsync with safe dereferencing of links + exclusions of all updates/ directories (crawler)
      • PR from Daniel to remove the “latest” : less data to copy \o/ - Stop generating `latest/` links directory in dynamic tiers by daniel-beck · Pull Request #747 · jenkins-infra/update-center2 · GitHub
      • TODO: trigger mirrorbits sync at the end (after the parallel steps). Requires a bit of Kubernetes.
    • WIP: mirrorbits ingress fix (nothing done except reproducing the issue locally)
  • ftp.belnet.be should be removed from mirrors or a fall-back offered
    • Still disabled
    • Can’t reproduce reporter’s network problem, hard to tell where the error is
    • Let’s contact mirror admins through the public support email
  • Proposal for application in publick8s to migrate to arm64
    • Knowledge sharing done (Stephane → Herve)
    • Migrated:
      • public Nginx ingress :party:
      • incremental-publisher
    • WiP:
      • Publishing arm64 for all the other Docker repositories
      • Some are not easy (rabbit hole):
        • uplink is using node 9 / node 10, need a bit of upgrade. Let’s check if there are node 9 /10 arm images.
      • next candidates to deploy:
        • (our own docker image) plugin-site-api
        • (our own docker image) mirrorbits (current)
        • (our own docker image) mirrorbits-parent (with httpd, rsync and mirrorbits)
        • cert-manager
        • datadog-cluster-agent
    • Next:
      • weekly.ci.jenkins.io => easy as controller only
    • ARM64 on other locations of the platform (future)?
      • privatek8s cluster: adding an arm64 node pool to be used by infra.ci for agents
      • VPN VM
    • Takeaway: It’s not only about shrinking cost (because arm64 is cheaper for the same workload), but also a cleanup of services \o/
  • “Mirror Status” link from get.jenkins.io returns a 404 error
    • TODO
  • Publishing Jenkins Security Scan rules as packages
    • To answer and close
  • track the new jdk21 version from adoptium
    • Knowledge sharing (Bruno + Stephane → Damien)
    • Next step: windows containers and tools installation
    • s390x tool installation will cherrypick Herve’s work from jenkinsci/docker images
  • Packer: GOSS version tracking and moving sanity check to goss
    • Linux’s asdf and nodejs and npm are checked (NPm is always “latest”)
    • Linux other tools: WiP PR - WIP goss transition by smerle33 · Pull Request #852 · jenkins-infra/packer-images · GitHub
    • Windows PR ready to review: tests(windows) Use `goss` for sanity checks for JDKs and trivy by dduportal · Pull Request #818 · jenkins-infra/packer-images · GitHub
    • Next: factorize and move all to goss
  • speed up the Docker image library to create/push tags at the same time for both GH and Docker (instead of running additional build)
    • TDD started
    • handover to do this week before Stephane is off
    • info: rolled back the “docker-confluence-data” (aka. wiki) tag discovery on infra.ci
  • Upgrade to Kubernetes 1.26
    • Changelog being read (50% done)
  • AWS: decrease cost for ~Summer~ Autumn 2023
    • Back to backlog (but 2 VMs to be migrated)
  • Matomo github/docker repos
    • No work done here
    • Gavin gave us details
    • Let’s pair on it
  • Ensure removed jenkins.io pages aren’t accessible and indexed anymore
    • Delayed from last week time to check backups
    • Planning for this Thursday (tomorrow is security advisory)
  • Planning for supported JDK versions in Jenkins Infrastructure
    • TODO: removing all JDK19 remainings
    • JEP 2+2+2 still under review
  • Retiring Chinese jenkins.io website:
    • Kevin and Damien need to sync to get started for last miles

• ToDo (next milestone) (infra-team-sync-2023-10-31 Milestone · GitHub)