Attendees 
- @dduportal (Damien Duportal)
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
- @krisstern (Kris Stern)
- @kmartens27 (Kevin Martens)
Announcements 
- ci.jenkins.io hammered by requests, might be a GSoc student (?)
- API is not expected to be available as per Daniel message yesterday (Apache blocks the
/api
) - Proposal to export JENKINS_HOME data (partial data, without credential)
- WDYT about daily export?
- Would be good to have it before last week of March, at least a “one time export” initially
- Daniel proposal is interesting: what about blocking all AWS/GCP/azure/Alibaba public Ip blocks from reaching ci.jio?
- Might have edge / few impacts but overall good protection from unlegitimate use cases
- Need an issue, and a brownout setup
- API is not expected to be available as per Daniel message yesterday (Apache blocks the
- Jenkins Weekly Releases
- Last week: 2.500 (security release) was succesfull last week (Wednesday)
- This Week: 2.501 started in time today
Upcoming Calendar 
- Next Weekly: 2.502, Tuesday 18, 2025
- Next LTS: 2.492.3, Wednesday April 2, Bruno is the release lead, mentored by Kris Stern
- Next baseline will be selected March 19
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 2025-03-17: infra.ci Azure VMs credential - [infra.ci.jenkins.io] Azure Service Principal Credential used to spawn Azure VM agents expires on `2025-03-17` · Issue #4588 · jenkins-infra/helpdesk · GitHub
- 2025-03-18:
- Azure File Storage credential to deploy stats.jio from infra.ci: [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy `stats.jenkins.io` expires the 2025-03-18 · Issue #4589 · jenkins-infra/helpdesk · GitHub
- Azure File Storage credential to deploy contributors.jio from infra.ci: [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy `contributors.jenkins.io` expires the `2025-03-18` · Issue #4590 · jenkins-infra/helpdesk · GitHub
- Azure File Storage credential to deploy plugins.jio from infra.ci: [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy `plugins.jenkins.io` expires the `2025-03-18` · Issue #4591 · jenkins-infra/helpdesk · GitHub
- 2025-03-23:
- Azure File Storage credential used by Terraform Backends + Cloudflare APi tokens: [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy `plugins.jenkins.io` expires the `2025-03-18` · Issue #4591 · jenkins-infra/helpdesk · GitHub
- Azure DNS credential used by Let’s Encrypt on trusted.ci and cert.ci: [cert.ci.jenkins.io, trusted.ci.jenkins.io] Azure Principal used for Let's Encrypt DNS challenges expires on `2025-03-23` · Issue #4593 · jenkins-infra/helpdesk · GitHub
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- December: $4,4k (invoice)
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $1486 (forecast at 4.4k)
-
Azure Sponsorship (Microsoft Credits) - Remaining: $45,973 until 31 May 2025
- December: $9,5k
- January: $13,1k
- February: $11.2k
- March: $1,691 (forecast at 4.5k)
- Cleanup of ci.jio done yesterday, too soon to evaluate
-
DigitalOcean - Remaining $14.1k until January 02, 2026
- December: $192 (invoice)
- January: $219 (invoice)
- February: $237 (invoice)
- March: $93 (forecast at $259)
-
AWS:
- CloudBees:
- December: $540
- January: $543
- February: $550
- March: $169 (forecast at $536)
- Sponsored account (~$51k credits lefts until 01/31/2027)
- December: $595
- January: $1.4k
- February: $8.5k
- March: $4,8k (forecast at $10.8k)
- CloudBees:
-
Jfrog Artifactory Usage
- Storage (sunday) we were < 4Tb (expected threshold is 5)
- Darin worked to decrease after last week “boost” to 6Tb
- Bandwidth still high though (> 10 Tb threshold)
- Storage (sunday) we were < 4Tb (expected threshold is 5)
Notes 
-
Done:
- Delete my account on accounts.jenkins.io.
- [infra.ci.jenkins.io] Netlify token expires 7 of March 2025
- [infra.ci.jenkins.io] NPM token expires the 7 March 2025
- CD not working with repo jenkins-test-harness
- Artifactory content is not correct
- [jenkins-infra] smerle33 to be set owner
- Windows agents failing to clone Git (LongPathsEnabled disabled)
- Stay on Git 2.47.1, 2.48.1 is buggy for long path
- Windows agents have different character set than in the past
- [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents
- [CD / GitHub Actions] Reached Docker.io rate limit
-
-
Support
- New mirror in India
- 2 of the 3 mirrors added succesfully last week.
- Waiting from them for the 3rd (looks like their server is having issues).
- Damien to write runbook to extend mirror management knowledge as underlined by Stephane
- Jenkins Jira license needs to be renewed
- Mark created a Linux Foundation issue
- Add a real-world job to weekly.ci.jenkins.io
- We should move this service out of publick8s AKS cluster to a tiny DO VM to allow this
- Deploy
jenkins-prototype
on Netlify- As the repository is now under jenkinsci GH Org, we need help from jenkinsci admin
- Tip: let’s ping jenkins-admin team in the issue to avoid Tim as a SPOF if he’s not available
- New repo
scoop-bucket
for app manifest distribution- Repo created (now Stephane is admin of jenkins-infra GH Org.)
- Valentin is testing initial release and setup. Waiting for feedback
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Still waiting for pairing with Andrew to handover the task
- KK’s data is flowing \o/
- https://updates.jenkins.io/stable/latest/jenkins.war returns 404
- Delayed
- CD release of
pipeline-maven
failed with 403- Delayed
- New mirror in India
-
Keep platform up to date/in shape
- Enhancements possible for packer-image pipeline
- done: Faster PRs (no image export by default)
- WiP: Faster PRs with conditional builds and labels to control builds
- [pipeline-library/updatecli] Allow custom version and execution in current “pipeline node” context
- First feature (custom version) is now working in draft PRs (1 non regression, 1 custom version). Currently in review (codestyle and cleanup)
- Second feature to get started (conditional node allocation)
- Enhancements possible for packer-image pipeline
-
Artifactory
- Reduce Artifactory storage and bandwidth use
- Meeting with Daniel, need to plan the jcenter-cache removal (implies backup before)
- [repo.jenkins-ci.org] Ensure repository
incrementals
is automatically garbage-collected- Still cleaned daily manually by Darin, need to automate with the team
- Reduce Artifactory storage and bandwidth use
-
- ci.jenkins.io slow response and slow loading of test results on startup
- We increase the controller size: no more I/O, more memory in the heap (14 → 30)
- LLM usages seems to cause most of the slow downs now, we have to update with IP blockers
- Build failed to resume
- Something wrong in EC2 plugin: bad config? plugin bug? Need to diagnose
- [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM
- Plugin BOM release builds on AWS ci.jenkins.io fail much more frequently
- docker-ssh-agent builds consistently timeout on ci.jenkins.io
- Also docker-inbound-agent, but not docker (controller), only on Windows tests
- It does not happen during releases because we don’t run tests
- Worth testing the tests on trusted.ci (in Azure)
- Instability of artifact-caching-proxy on AWS
- [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
- ci.jenkins.io slow response and slow loading of test results on startup
-
[INFRA-3100] Migrate updates.jenkins.io to another Cloud
- Delayed cleanup
-
[INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
- Delayed (no news from Philipp)
-
-
New Issues (to triage):
-
- Let’s start with kubectl and changelog
-
build failure with useArtifactCachingProxy=true and dependency with version range
-
External user struggling to submit story to stories.jenkins.io
-
ToDo (next milestone) (infra-team-sync-2025-03-18 Milestone · GitHub)