Infrastructure Team Meeting - May 06, 2025

[dduportal (Damien Duportal)]

2025 05 06 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @jayfranco999 (Jay Reddy) - out of office during the meeting
• @MarkEWaite (Mark Waite) - attending another meeting this week
• @poddingue (Bruno Verachten)
• @biru-codeastromer (Birajit)

Announcements

1. Previous meeting (2025-04-29) was cancelled
2. Jenkins Weekly Releases
   • 2 weeks ago: 2.507 succesfully released the 22 April 2025 with no issues - You're invited to talk on Matrix
   • Last week: 2.508, started 30 min late due to infra migration in progress but was succesful - You're invited to talk on Matrix
   • This week: 2.509, started on time - Wip: You're invited to talk on Matrix
3. Announcements:
   • OSUOSL is recruiting $250k of funding to continue their services (blog post)

Upcoming Calendar

• Next Weekly: 2025-05-13, 2.510
• Next LTS: 2025-05-28, 2.504.2, Kris Stern is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-05-12:
    • (Issue to be done) Digital Ocean PAT expires => @smerle
  • 2025-05-14:
    • (Issue to be done) [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) => @dduportal
  • 2025-05-22:
    • (Issue to be done) Azure SP used by updatecli in infra.ci: Extend Azure AD Application password validity for packer-resources (current end date: 2025-05-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #1030 · jenkins-infra/azure · GitHub
• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • February: $3.9k (invoice)
  • March: $4,372 (invoice)
  • April: $4,055 (waiting for final invoice)
  • May: $601 (forecast at $3.0k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $28,884 until 31 May 2025

  • February: $11.2k
  • March: $4,276
  • April: $12.1k
  • May: $2,390 (forecast at $12.4k)

• DigitalOcean - Remaining $14,300 until January 02, 2026

  • February: $237 (invoice)
  • March: $272 (invoice)
  • April: $349 (invoice)
  • May: $64.70 (forecast at $335)

• AWS:

  • CloudBees:

    • February: $550
    • March: $551
    • April: $532
    • May: $81 (forecast at $547)

  • Sponsored account (~$36,727 credits lefts until 01/31/2027)

    • February: $8.5k
    • March: $14,649
    • April: $1,977
    • May: $1 (forecast at $5)

• Jfrog Artifactory Usage

  • Storage: 3.65TB
    • jcenter cleanup: announced Friday, started Monday, in progress
  • Bandwidth:
    • March: 35.25 TB (better than expected)
    • April: 26.34 TB (good! Mostly ‘releases’ repo: 22.25Tb, then 1.21 Tb on jcenter and 1 Tb on incrementals)
    • May: 3.81 Tb (3.61 Tb is ‘releases’, forecast at ~20 TB)

Notes

Last milestone (2025-04-29 meeting cancelled)

• Done:

  • Support:
    • Mirror site is not accessible: https://mirror.bom.albony.in/
    • docker-ssh-agent builds consistently timeout on ci.jenkins.io
    • Build failed to resume
  • Keep the platform up to date/in good shape:
    • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2025-04-30
    • Deprecate ircbot service
    • [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents
    • Upgrade to Kubernetes 1.31
    • [infra.ci.jenkins.io] Updatecli: Use separated pipelines + organization scanning for all updatecli processes in infra.ci.jenkins.io controller

• Closed as not planned:

  • Several Jenkins core tests on ci.jenkins.io fail more often since transition to AWS

Current milestone

• Done:

  • Support:
    • Revert #4536
    • Lost admin permissions on ci.jenkins.io
    • Jenkins Updates for Plugins error in Belarus
  • Keep the platform up to date/in good shape:
    • [privatek8s] Migrate AKS cluster to the sponsored subscription
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.504.1
    • JDK patch upgrade campaign (April 2025)
    • Jenkins Controllers in Azure VMs: use workload identity management to allow managing Azure VM agents without credentials
      • [trusted.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents and Lets Encrypt
      • [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
    • Remove DockerHub “pull credentials” from Kubernetes clusters

• Closed as not planned:

  • Read timed out

• Work in Progress:

  • Support:

    • Add a real-world job to weekly.ci.jenkins.io
      • Delayed to June => @dduportal to comment and then backlog

  • Keep the platform up to date/in good shape:

    • Update Jira LTS from 9.12.x to 10.3.x
      • Mark opened to the LF. Assigned but no response yet.
      • No emergency (Sept/Oct. is EOL usually)
    • APT update fails with The following signatures were invalid: EXPKEYSIG 4528B6CD9E61EF26 Puppet, Inc. Release Key (Puppet, Inc. Release Key)
      • Not planned: let’s close? We want to focus on migrating away from puppet.
    • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
      • privatek8s done, back to backlog
      • ETA: before end of September 2025
    • [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
      • Back to backlog (still buggy provider)
    • Ubuntu 22.04 upgrade campaign
      • Stuck by puppet (in Azure/DO) and pkg.origin (AWS CloudBees)
      • Azure: https://techcommunity.microsoft.com/blog/linuxandopensourceblog/canonical-ubuntu-20-04-lts-reaching-end-of-standard-support/4392463
      • Resume running Puppet in a container with a production backup
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • Delayed to next milestone

  • 2025 Cloud Usage: ensure that we can run until end of year

    • [puppet.jenkins.io] Migrate to DigitalOcean

  • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature

    • KK had issues in the past 3 months. Should we add a monitor to help here?
    • Damien will meet with Andrew the 15th for a knowledge sharing session

  • Use JDK21 Platform-wide

    • Current scope: controller and agents JDK runtime (not the build JDKs)
    • Should be 1 big PR for ci.jio, trusted and cert at the same time (both controllers + agents). Should be the same for infra.ci and release.ci.
    • Move controllers to JDK21 (runtime)
    • Move agents to JDK21 (runtime)

• ToDo (next milestone) (GitHub · Where software is built)

  • Certbot: keep it up to date (along with its plugins)
  • Tombstone OSUOSL VMs edamame and lettuce
  • OSUOSL: decrease mirror dependency

• Issues delayed (after triage):

  • JDK25 issues:
    • JDK25 integration · Issue #4641 · jenkins-infra/helpdesk · GitHub
    • packer-images: use api.adoptium.net for updatecli and installation of JDKs · Issue #4640 · jenkins-infra/helpdesk · GitHub
    • back to triage
  • Chinese website
    • back to triage
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools