tags: meeting, infrastructure
project: infrastructure
2025-04-15
Zoom Meeting Room
Shared Calendar
Chat Room #jenkins-infra
Google Group (mailing list) jenkins-infra
Previous Meeting Notes
Attendees 
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
- @kmartens27 (Kevin Martens)
Announcements 
- Jenkins Weekly Releases
- Last Week: 2.505 succesfully released the 8 April 2025 with no issues - You're invited to talk on Matrix
- This week: 2.506, started on time -
- ci.jenkins.io runs in Azure with no problem
- Weekend restart was needed to complete the BOM build (issue resolved)
- Reminder: GitHub Milestones do not allow ordering of tasks: we must switch to GitHub projects to share priority
- Digital Ocean maintenance re-scheduled: Start: 2025-04-30 13:00 UTC End: 2025-04-30 21:00 UTC
- That is an LTS release day, will the DigitalOcean maintenance affect release?
Upcoming Calendar 
- Next Weekly: 2025-04-22, 2.507
- Next LTS: 2025-04-30, 2.504.1, Krist Stern is release lead, release candidate today
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 2025-04-30:
- Artifactory RPU token expires. Issue to create (last rotation: [Incident] Windows build of plugins don't start on `ci.jenkins.io` · Issue #4490 · jenkins-infra/helpdesk · GitHub)
- 2025-04-30:
- Next major event: N.A.
Cloud Budgets
-
Azure CDF:
- January: $4.3k (invoice)
- February: $3,9k (invoice)
- March: $4,372 (invoice)
- April: $1866 (forecast at $4,320)
- To be updated
-
Azure Sponsorship (Microsoft Credits) - Remaining: $39,039 until 31 May 2025
- January: $13,1k
- February: $11.2k
- March: $4,276
- April: $4,282 (forecast at $8,5k)
- ci.jenkins.io is clearly using
-
DigitalOcean - Remaining $14,395 until January 02, 2026
- January: $219 (invoice)
- February: $237 (invoice)
- March: $272 (invoice)
- April: $133 (forecast at $300)
-
AWS:
-
CloudBees:
- January: $543
- February: $550
- March: $551
- April: $227 (forecast at $473)
-
Sponsored account (~$36,727 credits lefts until 01/31/2027)
- January: $1.4k
- February: $8.5k
- March: $14,649
- April: $1,974 (forecast at $2,505 -83% from last month)
-
-
Jfrog Artifactory Usage
- Storage: 4.24TB (increased: jcenter cleanup will be more than needed)
- Darin has completed a back of jcenter
- Mark started a backup a few days ago
- Bandwidth:
- March: 35.25 TB (better than expected)
- April: 14.63 TB (forecast at 30 TB)
- Storage: 4.24TB (increased: jcenter cleanup will be more than needed)
Notes 
-
Done:
- [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
- Plugin Health Scoring release did not get a proper Docker Tag
- Very few maven-bom agents allocated on ci.jenkins.io
- [ci.jenkins.io]
ERR_CONNECTION_RESETor connection stuck when using IPv6 access - [ci.jenkins.io] Migrate controller VM back to Azure Sponsored Subscription
-
- update-center.json are lagging on mirrors.jenkins.io
- Jenkins Updates for Plugins error in Belarus
- Issue seems to be local to users in Belarus, not at the mirror site
- Chinese jenkins site incorrect site redirection
- Netlify does not allow Docker build in their site deployments
- APT update fails with
The following signatures were invalid: EXPKEYSIG 4528B6CD9E61EF26 Puppet, Inc. Release Key (Puppet, Inc. Release Key)- GPG signatures expired for those outdated Puppet versions
- Will need to use other alternatives to install
- [cert.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents
- Stephane started on the task, more work needed for the DNS certbot
- [puppet.jenkins.io] Migrate to DigitalOcean
- Migration to continue
- 2025 Cloud Usage: ensure that we can run until end of year
- Migrated ci.jenkins.io to Azure - major step to use Microsoft donation before it expires
- More migrations coming
- Upgrade to Kubernetes 1.31
- Upgrading by creating a new cluster using 1.31 in the sponsored account
- New cluster is a work in progress, migration done by creating the new cluster
- [Azure] Migrate (e.g. re-create) AKS clusters
publick8sandprivatek8swith modern settings (private API, Azure Linux, NAT outbound) - [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
- Add a real-world job to weekly.ci.jenkins.io
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Lower priority than our sponsorship activities
- [privatek8s] Migrate AKS cluster to the sponsored subscription
- Jenkins Controllers in Azure: use workload identity management to allow managing Azure VM / ACI agents without credential
- [infra.ci.jenkins.io] Updatecli: Use separated pipelines + organization scanning for all updatecli processes in infra.ci.jenkins.io controller
- Replace all Pipelines that use the parallel Pipeline that builds Docker and runs updatecli
- Segregate the scripts into separate Pipelines
- Check each repository that use the script
- Replace all Pipelines that use the parallel Pipeline that builds Docker and runs updatecli
-
- [trusted.ci.jenkins.io] Use Azure Workload Identity for Azure VM agents and Lets Encrypt
- [ci.jenkins.io] Run ci.jenkins.io and its agents on Java 21 instead of Java 17
- [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
- We will need to update our VM’s with explicit outbound access
- [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
- Move collection of stats out from Kohsuke’s home
- Phase 2 of the earlier item
- Support [skip ci] on default branch
- Create build for jenkinsci/winp on release ci server
- [Update Center] HTTP/404 on
/current/updates/*.json*links - dnf5 update fails with gpgcheck=1
- Add monitoring for CD secrets updates
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
-
ToDo (next milestone) (GitHub · Where software is built)