Attendees 
- @dduportal (Damien Duportal)
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
- @kmartens27 (Kevin Martens)
Announcements 
- Jenkins Weekly Releases
- Last Week: 2.501 released succesfully - You're invited to talk on Matrix
- This Week: 2.502 in progress - You're invited to talk on Matrix
- Announcements:
- Git 2.49.0:
- Windows LongPath bug still present
- Not available on Linux arm64 :man-facepalming:
- packer images builds are failing since we use package. Yet another reason to start building Git on Linux ourselves.
- Let’s enable GHA restriction on jenkins-infra org
- Git 2.49.0:
Upcoming Calendar 
- Next Weekly: 2.503, 25 March 2025
- Next LTS: 2.492.3, Wednesday April 2, Bruno is the release lead, mentored by Kris Stern
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 2025-03-23:
- Azure File Storage credential used by Terraform Backends + Cloudflare APi tokens: [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy `plugins.jenkins.io` expires the `2025-03-18` · Issue #4591 · jenkins-infra/helpdesk · GitHub
- Azure DNS credential used by Let’s Encrypt on trusted.ci and cert.ci: [cert.ci.jenkins.io, trusted.ci.jenkins.io] Azure Principal used for Let's Encrypt DNS challenges expires on `2025-03-23` · Issue #4593 · jenkins-infra/helpdesk · GitHub
- 2025-03-23:
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- December: $4,4k (invoice)
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $2,452 (forecast at 4.5k)
- Increase on publick8s: both VMs and outbound NAT gateway (not sure why)
-
Azure Sponsorship (Microsoft Credits) - Remaining: $45,132 until 31 May 2025
- December: $9,5k
- January: $13,1k
- February: $11.2k
- March: $2,534 (forecast at 4.3k)
- Cleanup of ci.jio: daily from ~$450 to ~$250
-
DigitalOcean - Remaining $14,6k until January 02, 2026
- December: $192 (invoice)
- January: $219 (invoice)
- February: $237 (invoice)
- March: $154 (forecast at $266)
-
AWS:
- CloudBees:
- December: $540
- January: $543
- February: $550
- March: $299 (forecast at $540)
- Sponsored account (~$51k credits lefts until 01/31/2027)
- December: $595
- January: $1.4k
- February: $8.5k
- March: $11,5k (forecast at $21k)
- Issue with Cost Explorer: need to contact AWS support (usage type is empty)
- A lot of snapshots
- Spot Instances
- BOM builds
- S3 artifact manager to decrease controller cost as well
- CloudBees:
-
Jfrog Artifactory Usage
- Storage: 4.12 TB
- Darin’s cleanup helps to keep us below the 5 TB threshold
- Next step would be jcenter removal. April?
- Bandwidth still high though (> 10 Tb threshold)
- March: 24.2 Tb
- A lot of variations: some days at 1 Tb, others at 280Gb
- March: 24.2 Tb
- Storage: 4.12 TB
Notes 
-
Done:
- Remove remoting component JIRA ticket default assignee
- [javadoc] build fails since removal of
org.kohsuke:github-api
from repo.jenkins-ci.org - [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy
plugins.jenkins.io
expires the2025-03-18
- [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy
contributors.jenkins.io
expires the2025-03-18
- [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy
stats.jenkins.io
expires the 2025-03-18 - [infra.ci.jenkins.io] Azure Service Principal Credential used to spawn Azure VM agents expires on
2025-03-17
- Add Bruno Verachten as a Jenkins LTS release lead
- Jenkins Jira license needs to be renewed
- ci.jenkins.io slow response and slow loading of test results on startup
- Plugin BOM release builds on AWS ci.jenkins.io fail much more frequently
- New mirror in India
- New repo
scoop-bucket
for app manifest distribution - [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
-
- [ci.jenkins.io] Use Spot instances for VM agents to decrease costs
- Reconsider regarding the cost spendings (NAT gateway)
- [cert.ci.jenkins.io, trusted.ci.jenkins.io] Azure Principal used for Let’s Encrypt DNS challenges expires on
2025-03-23
- Stephane
- [Terraform Backends] Expiration of credentials for Backend States and Cloudflare API tokens the
2025-03-23
- Damien
- [packer-images] logs within windows agents on amazon
- No changes
- [ci.jenkins.io] Fix HTTP/429 DockerHub errors for buildx builder using
docker-container
- No changes
- [ci.jenkins.io] Docker pull fails with random
ERROR: failed to read expected number of bytes: unexpected EOF
- Let’s try the ttl to zero on registry
- Otherwise we’ll need to wipe data weekly
- Build failed to resume
- No changes
- Enhancements possible for packer-image pipeline
- No changes
- Add a real-world job to weekly.ci.jenkins.io
- No changes
- We might want to block IPs like ci.jio on this one
- We want to move it to a VM in DigitalOcean
- https://updates.jenkins.io/stable/latest/jenkins.war returns 404
- No changes
- docker-ssh-agent builds consistently timeout on ci.jenkins.io
- No changes
- CD release of
pipeline-maven
failed with 403- deploy at the end to be merged
- Upgrade to Kubernetes 1.31
- kubectl is there in 1.31
- Next step: changelog and planning
- Instability of artifact-caching-proxy on AWS
- Scoped to BOM only builds
- Caching saved day (2M → 200k request for a full test BOM)
- Scaled down ACP to 1 replica => way better results
- Need to change from statefulset to a deployment to avoid availability issues
- Prior to Kube 1.31 upgrade on AWS
- [pipeline-library/updatecli] Allow custom version and execution in current “pipeline node” context
- “custom version” feature is ready to roll, reviewed in team, tested. Need to deliver it.
- “use the same node as context caller” is in progress. Tests are in progress, but initial results are good. Almost ready for review.
- [repo.jenkins-ci.org] Ensure repository
incrementals
is automatically garbage-collected- Delayed to April, currently done manually by Darin
- Reduce Artifactory storage and bandwidth use
- Kept open until sub issues are solved
- [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM
- Need to add a EKS cron pod to populate it weekly
- Deploy
jenkins-prototype
on Netlify- All accesses have been granted to Stephane. He can see the repo on Netlify now
- Next step: set up the website
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- No changes
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
- No changes
- [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
- No changes
- [ci.jenkins.io] Use Spot instances for VM agents to decrease costs
-
New Issues/Triage:
-
Delayed:
- Move collection of stats out from Kohsuke’s home
- Support [skip ci] on default branch
- Create build for jenkinsci/winp on release ci server
- [Update Center] HTTP/404 on
/current/updates/*.json*
links - dnf5 update fails with gpgcheck=1
- build failure with useArtifactCachingProxy=true and dependency with version range
- External user struggling to submit story to stories.jenkins.io
- Add monitoring for CD secrets updates
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
-
ToDo (next milestone) (infra-team-sync-2025-03-25 Milestone · GitHub)