Attendees 
- @dduportal (Damien Duportal)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
- @biru-codeastromer (Birajit)
- @jayfranco999 (Jay Reddy)
- @kmartens27 (Kevin Martens)
Announcements 
- Jenkins Weekly Releases
- Last Week: 2.506 succesfully released the 15 April 2025 with no issues - You're invited to talk on Matrix
- This week: 2.507, started on time -
- Team availabilities
- Damien is off this Friday (25 April)
- Stéphane is off next week (28 April → 02 May) and the Wednesday 7 May
- 1, 8 and 29 May are banking holidays for Bruno, Stephane and Damien as well
- Announcements:
- DigitalOcean Emergency Core Infrastucture today (Start: 2025-04-22 13:00 UTC End: 2025-04-22 19:00 UTC).
- No digitalocean API call (Terraform project jenkins-infra/digitalocean => new puppet controller and weekly.ci new VM, eventually archives.jenkins.io)
- Kubernetes Upgrade (1.30 → 1.31) on
publick8s
this Thurday 24 April (07:30am UTC, 09:30am Paris)- No objection: let’s roll
- Adoptium JDK trimestrial upgrade on the 4 JDKs we use (8,11,17 and 21): JDK patch upgrade campaign (April 2025) · Issue #4642 · jenkins-infra/helpdesk · GitHub
- JDK25 incoming
- Subject tracked by Bruno and Kris, helpdesk issue opened by Bruno, thanks!
- We have to:
- Add the JDK25 in agents (all in on image)
- Add the JDK25 in jenkins tools (all controllers)
- Pipeline library
- Timeline: ETA in May for infra
- JDK21 as runtime for controllers: [ci.jenkins.io] Run ci.jenkins.io and its agents on Java 21 instead of Java 17 · Issue #4623 · jenkins-infra/helpdesk · GitHub
- Ubuntu 20.04 is being dropped
- Ref. Ubuntu 22.04 upgrade campaign · Issue #2982 · jenkins-infra/helpdesk · GitHub
- Why not Ubuntu 24.04? Because of
curl
issue still present: Ubuntu 22.04 upgrade campaign · Issue #2982 · jenkins-infra/helpdesk · GitHub. 22.04 is supported until April 2027: we’re good for now. - GitHub Actions runners =>
ubuntu-latest
or pin to a version? => since it is only a GitHub action so nocurl
or noarm64
case - Puppet (6.x) controller machine (Azure and DO) with Puppet in a 20.04 container while using recent Ubuntu (22.04)?
- Shall we archive the Jenkins Infra Windows Agents Container Images?
- Not used anymore since [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents · Issue #4554 · jenkins-infra/helpdesk · GitHub
- It is an edge case of the pipeline-library/vars/buildDockerAndPublishImage.groovy at d2a0ec8feedbc138da11029ea4336116308f5853 · jenkins-infra/pipeline-library · GitHub pipeline library
- Less configuration and less builds
- ci.jenkins.io in Azure
- Let’s check GitHub · Where software is built
- BOM: we need support bundle (is plugin still there: no. Need to add it back.)
- Billing: looks good on Azure.
- AWS did sent us a form to apply for more sponsorship
- Mark and Damien work on this as it is very welcomed
- Incoming Software Update Campaigns (issues to create):
- JDK April patches (see above)
- Nginx Ingress 4.12 (ref. chore(updatecli) ping nginx-ingress to 4.11.x by dduportal · Pull Request #6354 · jenkins-infra/kubernetes-management · GitHub): grouping with Kube 1.31?
- Mirrorbits 0.5 → 0.6: issue to create, first bumped version since 6 years, could impact (but improve) both Update Center and get.jenkins.io
- Docker CE 28.0.x → 28.1.x: PR ready, will be applied to VMs when rebooting for this week upgrades
- ASDF 0.16.x: issue to create, PRs ready, should be an easy one (be careful: need tests before deploying)
- Deprecating docker-builder (websites) in favor of the all in one image?
- JQ 1.7: issue to create, mostly impacts report generations, UC and RPU
- Chocolatey 2.x in the all in one image
- DigitalOcean Emergency Core Infrastucture today (Start: 2025-04-22 13:00 UTC End: 2025-04-22 19:00 UTC).
Upcoming Calendar 
- Next Weekly: 2025-04-29, 2.508
- Next LTS: 2025-04-30, 2.504.1, Kris Stern is release lead
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 2025-04-30:
- Artifactory RPU token expires. Issue to create (last rotation: [Incident] Windows build of plugins don't start on `ci.jenkins.io` · Issue #4490 · jenkins-infra/helpdesk · GitHub)
- 2025-05-10:
- Azure AD Application password for Azure VM agents in trusted.ci.jenkins.io expires on 2025-05-10 (Azure AD Application password for Azure VM agents in `trusted.ci.jenkins.io` expires on `2025-05-10T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1009 · jenkins-infra/azure · GitHub)
- Note: should be closed in favor of [trusted.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents and Lets Encrypt · Issue #4630 · jenkins-infra/helpdesk · GitHub
- Azure AD Application password for Azure VM agents in trusted.ci.jenkins.io expires on 2025-05-10 (Azure AD Application password for Azure VM agents in `trusted.ci.jenkins.io` expires on `2025-05-10T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1009 · jenkins-infra/azure · GitHub)
- 2025-05-12:
- Digital Ocean PAT expires -
- [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) -
- 2025-04-30:
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $4,372 (invoice)
- April: $2,884 (forecast at $4,1k)
-
Azure Sponsorship (Microsoft Credits) - Remaining: $36,102 until 31 May 2025
- January: $13,1k
- February: $11.2k
- March: $4,276
- April: $7,276 (forecast at $9.9k)
-
DigitalOcean - Remaining $14,300 until January 02, 2026
- January: $219 (invoice)
- February: $237 (invoice)
- March: $272 (invoice)
- April: $229 (forecast at $315)
-
AWS:
-
CloudBees:
- January: $543
- February: $550
- March: $551
- April: $367 (forecast at $510)
-
Sponsored account (~$36,727 credits lefts until 01/31/2027)
- January: $1.4k
- February: $8.5k
- March: $14,649
- April: $1,976 (forecast at $1,998 -86% from last month)
-
-
Jfrog Artifactory Usage
- Storage: 3.76TB (decreased: a manual cleanup was most probably performed by Darin: thanks!)
- jcenter cleanup (to make this sustainable) both Darin and Mark completed a backup of jcenter
- Bandwidth:
- March: 35.25 TB (better than expected)
- April: 19.63 TB (forecast at 27 TB)
- Storage: 3.76TB (decreased: a manual cleanup was most probably performed by Darin: thanks!)
Notes 
-
Done:
-
-
Support:
- Jenkins Updates for Plugins error in Belarus
- Chinese jenkins site incorrect site redirection
- Delaying to May
- Add a real-world job to weekly.ci.jenkins.io
- Delayed to May (Stephane will be off next week)
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Delayed to next week (Damien is busy this week)
-
Keep Platform Up to date/in good shape:
- [infra.ci.jenkins.io] Updatecli: Use separated pipelines + organization scanning for all updatecli processes in infra.ci.jenkins.io controller
- A fix was needed on the pipeline library to support PHS
- need unit tests
- Thanks @alecharp!
- Some releases regressions to be checked (docker-confluence-data for instance)
- Should we turn automatic semantic version on by default?
- Need to deprecate the old “Docker && Updatecli” library
- Let’s also use this “sweep” to integrate Herve’s changes and the ubuntu-latest for release drafter failing GH workflows
- A fix was needed on the pipeline library to support PHS
- Upgrade to Kubernetes 1.31
- publick8s this Thursday?
- Shall we upgrade privatek8s?
- yes! Tomorrow
- Jenkins Controllers in Azure: use workload identity management to allow managing Azure VM / ACI agents without credential
- [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
- Agents OK
- Need to do Let’s Encrypt now: delayed to May.
- [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
- APT update fails with
The following signatures were invalid: EXPKEYSIG 4528B6CD9E61EF26 Puppet, Inc. Release Key (Puppet, Inc. Release Key)
- Damien to resume work this week
- [Azure] Migrate (e.g. re-create) AKS clusters
publick8s
andprivatek8s
with modern settings (private API, Azure Linux, NAT outbound)- WiP: see above for privatek8s
- [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- Delayed to May (not top priority and provider still buggy)
- [infra.ci.jenkins.io] Updatecli: Use separated pipelines + organization scanning for all updatecli processes in infra.ci.jenkins.io controller
-
2025 Cloud Usage: ensure that we can run until end of year
- [puppet.jenkins.io] Migrate to DigitalOcean
- [privatek8s] Migrate AKS cluster to the sponsored subscription
- Resuming work this week, next step is the private network access to allow Kubernetes management from infra.ci (first for terraform resources, then helmfile)
-
-
-
Next milestone:
- JDK patch upgrade campaign (April 2025)
- JDK25 integration
- [trusted.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents and Lets Encrypt
- @smerle on the Agents part (same as cert.ci). LE is delayed in May.
-
Delayed to May (or later):
- [ci.jenkins.io] Run ci.jenkins.io and its agents on Java 21 instead of Java 17
- [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
- [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
- Move collection of stats out from Kohsuke’s home
- Support [skip ci] on default branch
- Create build for jenkinsci/winp on release ci server
- [Update Center] HTTP/404 on
/current/updates/*.json*
links - dnf5 update fails with gpgcheck=1
- Add monitoring for CD secrets updates
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
- Ubuntu 22.04 upgrade campaign
-
-
ToDo (next milestone) (GitHub · Where software is built)