Attendees 
- @dduportal (Damien Duportal)
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
Announcements 
- Jenkins Weekly Releases
- Last Week: 2.503 was released successfully (package failed due to OSUOSL slowness: finished manually) - You're invited to talk on Matrix
- This Week: 2.504 is delayed to tomorrow (Wednesday April 2) as it will be part of to security advisory - https://groups.google.com/g/jenkinsci-advisories/c/gjEkmc02cU0
- Security Advisory tomorrow on Core releases (both weekly 2.504 and LTS 2.492.3) - https://groups.google.com/g/jenkinsci-advisories/c/gjEkmc02cU0
- Proposed 2.504 as next LTS baseline, agreed by Tim Jacomb
- 2025 Cloud Usage: we might not reach the end of year if neither Amazon and Microsoft renew their sponsorships.
- Tracked in 2025 Cloud Usage: ensure that we can run until end of year · Issue #4618 · jenkins-infra/helpdesk · GitHub
- Short term proposal: let’s move ci.jenkins.io back to Azure until end of May. It will delay usage of AWS credits and allows us to go up to end of September
- Other proposals in the issue to decrease spending in Azure CDF to anticipate after May
- We should start searching for another sponsor(s) (better safe than sorry)
Upcoming Calendar 
- Next Weekly:
- 2025-04-02: 2.504 tomorrow (security advisory)
- 2025-04-08: 2.505
- Next LTS:
- 2025-04-02: 2.492.3 (security advisory) - Bruno V. leads, assisted by Kris S.
- Proposed 2.504 as next LTS baseline, agreed by Tim Jacomb
- Next Security Release as per jenkinsci-advisories: https://groups.google.com/g/jenkinsci-advisories/c/gjEkmc02cU0
- Upcoming credentials expirations (~3 weeks):
- 2025-04-08:
- Azure SP for www.jenkins.io on trusted.ci.jenkins.io (to deploy website) - Azure File Share Principal `www.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #964 · jenkins-infra/azure · GitHub
- Azure SP for javadoc.jenkins.io on trusted.ci.jenkins.io (to deploy website) - Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #963 · jenkins-infra/azure · GitHub
- Azure SP for GeoIP (production) storage on publick8s (to mount Azure File volume) - Azure File Share Principal `geoip` on `publick8s` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #961 · jenkins-infra/azure · GitHub
- Azure SP for docs.jenkins.io on infra.ci.jenkins.io (to deploy website) - New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: 2025-04-08T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #960 · jenkins-infra/azure · GitHub
- 2025-04-15:
- Azure SP for Azure VM agents on cert.ci.jenkins.io - Azure AD Application password for Azure VM agents in `cert.ci.jenkins.io` expires on `2025-04-15T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #967 · jenkins-infra/azure · GitHub
- Worth moving to VM system identity and go credential-less (to test on ci.jio)
- Azure SP for Azure VM agents on cert.ci.jenkins.io - Azure AD Application password for Azure VM agents in `cert.ci.jenkins.io` expires on `2025-04-15T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #967 · jenkins-infra/azure · GitHub
- 2025-04-18:
- 2025-04-08:
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- December: $4,4k (invoice)
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $4,272 (forecast at 4.372k)
- $1,480 in VMs (spot and non spots)
- $657 in NAT gateway
- $635 in outbound bandwidth
- $585 in DNS
- $481 in Storage
-
Azure Sponsorship (Microsoft Credits) - Remaining: $43,358 until 31 May 2025
- December: $9,5k
- January: $13,1k
- February: $11.2k
- March: $4,276
- $1,931 in VMs (non spots)
- $825 in Redis cache
- $759 in NAT gateway
- $450 in storage
-
DigitalOcean - Remaining $14,528 until January 02, 2026
- December: $192 (invoice)
- January: $219 (invoice)
- February: $237 (invoice)
- March: $272 (invoice)
-
AWS:
- CloudBees:
- December: $540
- January: $543
- February: $550
- March: $551
- Sponsored account (~$28,265 credits lefts until 01/31/2027)
- December: $595
- January: $1.4k
- February: $8.5k
- March: $14,649
- Past 10 days were below $200 daily (~4,5 months if we keep this rate)
- Past 10 days were below $200 daily (~4,5 months if we keep this rate)
- CloudBees:
-
Jfrog Artifactory Usage
- Storage: 3.72 TB (steady)
- Bandwidth:
- March: 35.25 TB (better than expected)
- Last 14 days were at 11 Tb which is better thanks to Darin’s work: we can expect < 30 Tb next month at this rate
- March: 35.25 TB (better than expected)
Notes 
-
Done:
- Support:
- ci.jenkins.io:
- [ci.jenkins.io] Use EC2 local store (local NVMe instead of EBS network storage)
- [ci.jenkins.io] Windows agents - Docker fails with
error during connect: this error may indicate that the docker daemon is not running
- [ci.jenkins.io] Docker pull fails with random
ERROR: failed to read expected number of bytes: unexpected EOF
- [ci.jenkins.io] Fix HTTP/429 DockerHub errors for buildx builder using
docker-container
- [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM
- Archive jenkinsci/cloudbees-enterprise-plugins-plugin repository
- Deprecate the GitHub Pull Request Builder plugin
- Deploy
jenkins-prototype
on Netlify - ERR_CERT_COMMON_NAME_INVALID on https://ci.jenkins-ci.org
- archives.jenkins.io:
- build failure with useArtifactCachingProxy=true and dependency with version range
- ci.jenkins.io:
- Keep platform up to date:
- Cloud Costs:
- [aws.ci.jenkins.io] High EC2 costs on the
USE2-NatGateway-Bytes
andUSE2-DataTransfer-Regional-Bytes
- [ci.jenkins.io] Use Spot instances for VM agents to decrease costs
- Azure !
- EC2 plugin bug (known since november 2024): EC2 ephemeral agent takes a 10 min delay before detecting agent reclaim
- [ci.jenkins.io] Use Spot instances for VM agents to decrease costs
- [Azure/GeoIP] Cleanup unused staging volume
- [aws.ci.jenkins.io] High EC2 costs on the
- Update Center
- Support:
-
- Infra Tooling
- [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
- Buildx does not need imporvement
- Testing will happen on the jenkins-infra/docker-404
- WiP on the groovy code now
- Enhancements possible for packer-image pipeline
- Back to backlog
- [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
- Back to backlog (spam amount: still 1 or 2 a day. High enough to care, low enough not to prioritize for now)
- Delayed up to June (resume once we are Ok on the cloud billing)
- [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
- Support:
- S3 data dump of selected ci.jenkins.io data for GSoC 2025 LLM project
- Back to backlog, eventually look in May but no commitment
- Mark might have data to help them
- Several Jenkins core tests on ci.jenkins.io fail more often since transition to AWS
- Back to backlog until we moved ci.jio back to Azure
- Build failed to resume
- Back to backlog until we moved ci.jio from to Azure to EC2 (has it is a specific EC2 issue)
- Add a real-world job to weekly.ci.jenkins.io
- Delayed until we can move it to a DigitalOcean VM. keeping in milestone for cost reduction
- Might be done with the puppet.jenkins.io VM migration from Azure CDF to DigitalOcean
- docker-ssh-agent builds consistently timeout on ci.jenkins.io
- Not fixed by NVMe usage in VMs, neither by disabling Docker mirror registry
- Back to backlog until we moved ci.jio back to Azure
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Nothing done, need to contact Andrew. Decreased priority.
- S3 data dump of selected ci.jenkins.io data for GSoC 2025 LLM project
- Keep platform up to date:
- [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- 5.2.0 provider released, should fix the token issue, but users complains it does not
- WiP: using 5.x provider on the public terraform project
- Upgrade to Kubernetes 1.31
- kubectl OK, changelog checked: no breakage expected (Hervé also validated the “agent” cases on 1.31 and 1.32).
- See new issues to triage which impacts this one:
- No need to upgrade ci.jio EKS cluster as we move back to azure. Let’s start with new Kube 1.31 from scratch.
- privatek8s might need to be re-created. Same pattern.
- infraci-agents-1 AKS cluster this week (Thursday).
- [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- Update Center
- https://updates.jenkins.io/stable/latest/jenkins.war returns 404
- Back to backlog until mid April
- https://updates.jenkins.io/stable/latest/jenkins.war returns 404
- Reduce Artifactory storage and bandwidth use
- [repo.jenkins-ci.org] Ensure repository
incrementals
is automatically garbage-collected - Back to backlog until May but we keep monitoring weekly (and react if things goes bad)
- [repo.jenkins-ci.org] Ensure repository
- Infra Tooling
-
new Issues/to triage:
- Add to milestone:
- 2025 Cloud Usage: ensure that we can run until end of year
- [ci.jenkins.io] Migrate controller VM back to Azure Sponsored Subscription
- [ci.jenkins.io-agents] cluster migration to azure in kubernetes 1.31
- [puppet.jenkins.io] Migrate to DigitalOcean
- [Azure] Migrate (e.g. re-create) AKS clusters
publick8s
andprivatek8s
with modern settings (private API, Azure Linux, NAT outbound)
- 2025 Cloud Usage: ensure that we can run until end of year
- Keep in “triage”:
- [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
- [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
- Move collection of stats out from Kohsuke’s home
- Support [skip ci] on default branch
- Create build for jenkinsci/winp on release ci server
- [Update Center] HTTP/404 on
/current/updates/*.json*
links - dnf5 update fails with gpgcheck=1
- External user struggling to submit story to stories.jenkins.io
- Add monitoring for CD secrets updates
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
- Add to milestone:
-
ToDo (next milestone) (GitHub · Where software is built)