Attendees 
- @dduportal (Damien Duportal)
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
Announcements 
-
Jenkins Weekly Releases
- Last Week:
- 2.560 as usual (as per last team meeting)
- 2.561 as emergency release to ensure Fix triggering pipeline with parameters by timja · Pull Request #26685 · jenkinsci/jenkins · GitHub is delivered
- Issue with the Docker controller image which required many rebuilds due to Windows problems (agents stuck)
- This Week:
- 2.562 has been successfully released
- Issue with the Docker controller image which required many rebuilds due to Windows problems (agents stuck) AND network issues (GitHub outage aftermath)
- WAR is only signed by GPG now (no more jarsigner anymore)
- MSI is signed by LF certificate (ref. [pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026 · Issue #4923 · jenkins-infra/helpdesk · GitHub)
- Mark provides screenshots as it looks like there still might be issue on Windows. Delaying backports until we understand the new behavior better.
- 2.562 has been successfully released
- Last Week:
-
Announcements:
- Jenkins Digicert Code Signing Certificate expires in May: [pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026 · Issue #4923 · jenkins-infra/helpdesk · GitHub
- 2.561 was ~successful~ to be confirmed as Mark got Windows message in the setup wizard. Need to investigate to understand better.
- Next critical step: backporting to LTS line to ensure 2.555.2 MSI is also signed with LF certificate
- Need to communicate: changelog of 2.562 and 2.555.2 at least.
- Blog post?
- Medium term: move Windows environment (build AND signing) to our all-in-one VM (faster build, less maintenance)
- Issue to create as it is a distinct topic
- Long term: add OIDC on release.ci with a public issuer endpoint to allow credential-less signing with LF (safer for them)
- Issue to create as it is a distinct topic
- Incoming updates:
- (Critical) Incoming Java patch campaign
- JDK patch upgrade campaign (April 2026) · Issue #5081 · jenkins-infra/helpdesk · GitHub
- Temurin not available yet on our usual platforms
- (Important) Node JS 24.x
- Bump NodeJS from 22 to 24 (most recent LTS) · Issue #5080 · jenkins-infra/helpdesk · GitHub
- packer image is on 22, need to be bumped
- GHA are deprecating NodeJS 20 actions (security-scan, CD releases, and helpdesks)
- (Important) Docker CE 29+
- Bump Docker CE from 28.x to 29.x · Issue #5079 · jenkins-infra/helpdesk · GitHub
- Should fix Windows issues
- If it breaks plugins CI builds, then they must bump their
docker-clientAPI version - ATH is already using 29.x in their nested Docker engines
- (Nice to have) Maven 3.9.15 - see issue in milestone.
- (Nice to have) Helm v4
- Issue to create
- (Critical) Incoming Java patch campaign
- Team capacity:
- Herve is off this week
- Friday 1st May is banking day for everyone in France and India (at least!)
- Infra Roadmap: Jenkins Roadmap
* Damien to update for Feb/March/April/May- Current topics:
- MSI signing renewal => top priority until LTS backport is performed
- Azure Sponsored Subscription 2026: utilizes the 100k$ credits for the Jenkins Infrastructure
- trusted.ci controller (billing + 3 outages in the Azure VM hypervisor hosting it)
- Issue to create
- cert.ci controller
- Issue to create
- privatek8s: ~30$ daily!
- Issue to create
- trusted.ci controller (billing + 3 outages in the Azure VM hypervisor hosting it)
- Windows 2025/2022/2019
- Maven 4 RC/GA
- Proposed priorities:
- stats.jenkins.io stale data
- Get rid of puppet
- Current topics:
- Jenkins Digicert Code Signing Certificate expires in May: [pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026 · Issue #4923 · jenkins-infra/helpdesk · GitHub
Upcoming Calendar 
- Next infra meeting: 2026-05-05
- Next Weekly: 2026-05-05 - 2.563
- Next LTS: 2026-05-13 - 2.555.2 - Release Lead: Mark Waite
- RC: 2026-04-29
- Targeting new MSI signing
- Note: Infra team to update the release checklist as per their feedbacks. 2 focus points: timing and release.ci build (parsing/failing/parent/child)
- Next Security Release as per jenkinsci-advisories: 2026-04-29 (tomorrow), plugins only
- Upcoming credentials expirations (~3 weeks):
- 2026-05-01: Azure Credential used by Packer Image builds (issue to be created) => Damien + Jay
- 2026-05-13: NPM token used by infra.ci.jenkins.io (issue to be created)
- 2026-05-14: Netlify token used by infra.ci.jenkins.io (issue to be created)
- 2026-05-15: Azure File Share Credentials for (contributors|stats|plugins).jenkins.io websites used by infra.ci.jenkins.io (issue to be created)
- Next major event:
- cdCon May 18-20 in Minesota, Mark will be controller of ceremonies!
Cloud Budgets
-
Azure (CDF paid)
- February: $6.4k (invoice)
- March: $5.6k (invoice)
- April: $3.3k (forecasted at $3.65k)
- Cost reduction effort is paying
- Additional cost reductions are still ongoing
- CDF got the information that we should reach ~30k for 2026
-
Azure Sponsored subscription (Microsoft Credits): ~$94465 credits left until 20 Oct. 2027
- March: $1847 (invoice)
- April: $5.1k (forecasted at $5.7k)
- ~15 months left at current rate
-
DigitalOcean - Remaining $8,544.70 until January 02, 2027
- February: $565.03 (invoice)
- March: $423.63 (invoice)
- April: $775.30 (forecasted at ~$830)
- ~10 months left at current rate
-
AWS:
- Sponsored account, credits left: $13,880.67 until 05/31/2027, and $60k until 01/31/2028
- February: $11.1k (invoice)
- March: $14.0k (invoice)
- April: $13.7k (forecasted at $15.2k)
- Still high and increasing (EC2 plugin bug, BOM, non spots, old instance family, bandwidth)
- Sponsored account, credits left: $13,880.67 until 05/31/2027, and $60k until 01/31/2028
-
JFrog Artifactory Usage
- Storage: 1.83 Tb
- Steady
- Bandwidth:
- February: 17.46 Tb
- March: 27.24 Tb
- April: 43.36 Tb (forecasted at ~45 Tb)
- Abuse, abuse and abuse
- Mark/Damien needs to warn JFrog about the heavy use in March and April
- Mark needs to review the recent log analysis
- Storage: 1.83 Tb
Notes 
-
Done:
- uncategorized
- support
- keep-infra-sane-and-maintainable
-
- keep-infra-up-to-date
- support
- keep-infra-sane-and-maintainable
- [release.ci.jenkins.io] Avoid filling the JENKINS_HOME
- Need to resume
- [Uplink,Rating,PHS,Keycloak] Postgres version
13will lose standard Azure support on March 31, 2026. Upgrade now to avoid extended support charges starting August 1, 2026.- Need to resume
- Azure Sponsored Subscription 2026: utilizes the 100k$ credits for the Jenkins Infrastructure
- trusted.ci controller, cert.ci controller and privatek8s cluster
- Move docker controller images publication job from trusted.ci.jenkins.io to release.ci.jenkins.io
- Need to resume
- [(status|contributors|stats|docs|stories|cn|jenkins-io-components).jenkins.io, Websites PR previews] Netlify billing plan changed: evaluate and act
- Need to resume (backlog)
- Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io / cert.ci.jenkins.io)
- Datadog on publick8s is now collecting metrics from build reports
- WiP: some metrics are not collected, investigation
- cert.ci cannot publish build reports to storage.
- Need to set up a private endpoint.
- Next step:
- Integrate datadog monitors for the predefined list of job once metrics are fixed (and eventually setup alerts)
- Datadog on publick8s is now collecting metrics from build reports
- [release.ci.jenkins.io] Avoid filling the JENKINS_HOME
-
Issues staying in backlog/triage:
-
msbuild not available on ci.jenkins.io Windows agent for winp build process
-
[Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
-
[Azure] Merge webservices data storage accounts into a single one with NFS v4.1
-
[updates.jenkins.io] set up mirrorbits to keep serving update-center from mirrors even if outdated
-
private docker image registry for staging core security releases
-
support
-
keep-infra-sane-and-maintainable
-
Issues added to the next milestone: