Attendees 
- @dduportal (Damien Duportal)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @kmartens27 (Kevin Martens)
Announcements 
- Jenkins Releases
- Last Week (Security Advisory):
- 2.492.3 LTS released with success.
- 2.504 weekly released with success. Packaging jobs had to be aborted due to OSUOSL slowness during rsync, like 2.503 => to be watched on 2.505
- This Week:
- 2.505 started on time - You're invited to talk on Matrix
- Late revert by Mark Waite trying to avoid a change that breaks tests in plugin BOM
- Later fix by Damien Duportal to fix a Pipeline syntax error
- 2.505 started on time - You're invited to talk on Matrix
- Last Week (Security Advisory):
- ci.jenkins.io migration from AWS to Azure: 2025-04-09 (tomorrow) at 09:00am UTC
- Puppet GPG key expired: we cannot install puppet agent 6 or 7 anymore unless disabling GPG on their repository
- Ref. APT update fails with `The following signatures were invalid: EXPKEYSIG 4528B6CD9E61EF26 Puppet, Inc. Release Key (Puppet, Inc. Release Key) <release@puppet.com>` · Issue #4631 · jenkins-infra/helpdesk · GitHub
- Vagrant: fixed
- Current VMs: only a warning message
- New VMs: cloudinit to fix. Host the DEB package (
puppet-agent
for Bionic, Focal and Jammy) ourselves? - Puppet controller: shall we try to install a new one in that case?
- Warning about JDK17 EOL in 1 year
- Ref. [ci.jenkins.io] Run ci.jenkins.io and its agents on Java 21 instead of Java 17 · Issue #4623 · jenkins-infra/helpdesk · GitHub
- Proposal to move the Docker image “latest” to JDK21: Use JDK 21 as default JDK instead of JDK 17 by MarkEWaite · Pull Request #2008 · jenkinsci/docker · GitHub (ci.jenkins.io should use
lts-jdk17
so we are safe) - We already have many issues about this:
- Move agents to JDK21 (runtime) · Issue #4121 · jenkins-infra/helpdesk · GitHub
- Move controllers to JDK21 (runtime) · Issue #4123 · jenkins-infra/helpdesk · GitHub
- Switch default JDK to 21 for build tools · Issue #4125 · jenkins-infra/helpdesk · GitHub
- Switch default JDK to 21 for pipeline libraries · Issue #4126 · jenkins-infra/helpdesk · GitHub
- GitHub · Where software is built
- Team member days off:
- Stephane off this week: from 9 (tomorrow) back Monday (14)
- Damien is off next week (back on the office the 22)
- We need someone to lead the team meeting => @smerle
- We need someone to ensure weekly release Docker image tag is created and pushed => @mark
- Monday 21 April is banking day (Easter) in most of our countries
- Note: Chinese website Jenkins will be back(?)
- Let’s move it to Netlify so the SIG doc can iterate with autonomy on it (until it is ready to redeploy)
- DigitalOcean Core Infrastructure Maintenance 2025-04-14
- Status to add just in case
- GitHub Milestones do not allow ordering of tasks: we must switch to GitHub projects to share priority
Upcoming Calendar 
- Next Weekly: 2025-04-15, release 2.506
- Next LTS: 2025-04-30, 2.504.1, Krist Stern is release lead
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 2025-04-15:
- Azure SP for Azure VM agents on cert.ci.jenkins.io - https://github.com/jenkins-infra/azure/pull/967 => moving to workload identity instead (ref. [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents · Issue #4629 · jenkins-infra/helpdesk · GitHub)
- 2025-04-30:
- Artifactory RPU token expires. Issue to create (last rotation: [Incident] Windows build of plugins don't start on `ci.jenkins.io` · Issue #4490 · jenkins-infra/helpdesk · GitHub)
- 2025-04-15:
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $4,372 (invoice)
- April: $997 (forecast at $4,075)
-
Azure Sponsorship (Microsoft Credits) - Remaining: $42,370 until 31 May 2025
- January: $13,1k
- February: $11.2k
- March: $4,276
- April: $1,005 (forecast at $3.7k)
-
DigitalOcean - Remaining $14,528 until January 02, 2026
- January: $219 (invoice)
- February: $237 (invoice)
- March: $272 (invoice)
- April: $75 (forecast at $280)
-
AWS:
- CloudBees:
- January: $543
- February: $550
- March: $551
- April: $115 (forecast at $469)
- Sponsored account (~$36,726 credits lefts until 01/31/2027)
- January: $1.4k
- February: $8.5k
- March: $14,649
- April: $1397 (forecast at $6,430)
- CloudBees:
-
Jfrog Artifactory Usage
- Storage: 3.83 TB (slight increase from last week)
- Bandwidth:
- March: 35.25 TB (better than expected)
- April: 7.43 TB (forecast at 28 TB)
Notes 
-
Done:
- Support
- Keep Infra up to date
- 4 credential rotations, no issue though
- Improve Infra/Decrease Support
- ci.jenkins.io to Azure:
-
- 2025 Cloud Usage: ensure that we can run until end of year
- [ci.jenkins.io] Migrate controller VM back to Azure Sponsored Subscription
- Tomorrow (9 April)
- [privatek8s] Migrate AKS cluster to the sponsored subscription
- WiP on the network part (azure Net)
- Linked to Kubernete 1.31
- Linked to “improve AKS clusters to avoid depreciation” (private API, new outbound explicit NAT, new CNI)
- [puppet.jenkins.io] Migrate to DigitalOcean
- VM created and started
- Next step: Puppet bootstrap (first, agent using current agent, then new controller)
- [ci.jenkins.io] Migrate controller VM back to Azure Sponsored Subscription
- Keep Infra up to date
- [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- Delayed, because provider 5.x still buggy
- Upgrade to Kubernetes 1.31
- EKS: nothing to do
- (.*)-agents-1: Done (AKS)
- Next: privatek8s
- [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
- WIP on the pipeline Library, Bakefile is done, most of the groovy code is ready
- Testing in progress on the docker-404
- [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- Improve Infra/Decrease Support
- Jenkins Controllers in Azure: use workload identity management to allow managing Azure VM / ACI agents without credential
- [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
- Jenkins Azure VM part is almost done (1 PR away)
- Then: Let’s Encrypt with certbot
- [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
- [Azure] Migrate (e.g. re-create) AKS clusters
publick8s
andprivatek8s
with modern settings (private API, Azure Linux, NAT outbound)- privatek8s in progress, to also comply with kube 1.31 campaign and credits
- Jenkins Controllers in Azure: use workload identity management to allow managing Azure VM / ACI agents without credential
- Support
- Add a real-world job to weekly.ci.jenkins.io
- Need a new VM in DigitalOcean
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Delayed
- Add a real-world job to weekly.ci.jenkins.io
- 2025 Cloud Usage: ensure that we can run until end of year
-
New Issues (to triage):
- To next milestone:
- APT update fails with
The following signatures were invalid: EXPKEYSIG 4528B6CD9E61EF26 Puppet, Inc. Release Key (Puppet, Inc. Release Key)
- Proposal: host the 4 puppet-agent packages into our own infra and use them
- Chinese jenkins site incorrect site redirection
- Will be delayed, but we will propose the move to netlify on short term
- [trusted.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents and Lets Encrypt
- Same as cert.ci => should be an easy one
- [ci.jenkins.io] Run ci.jenkins.io and its agents on Java 21 instead of Java 17
- Will be closed as duplicate (of the other existing issue about JDK21 on Controller)
- Move controllers to JDK21 (runtime)
- APT update fails with
- Delayed:
- [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
- [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
- Move collection of stats out from Kohsuke’s home
- Support [skip ci] on default branch
- Create build for jenkinsci/winp on release ci server
- [Update Center] HTTP/404 on
/current/updates/*.json*
links - dnf5 update fails with gpgcheck=1
- External user struggling to submit story to stories.jenkins.io
- Add monitoring for CD secrets updates
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move agents to JDK21 (runtime)
- To next milestone:
-
ToDo (next milestone) (GitHub · Where software is built)