Infrastructure Team Meeting - June 10, 2025

dduportal · June 16, 2025, 8:04am

Attendees

@dduportal (Damien Duportal)
@jayfranco999 (Jay Reddy)
@smerle33 (Stéphane Merle)
@poddingue (Bruno Verachten)
@kmartens27 (Kevin Martens)

Announcements

Jenkins Weekly Releases
- Last Week: 2.513 was released - You're invited to talk on Matrix
  - Note: Docker image was delayed of ~4 hours due to DockerHub issues (HTTP/301 when trying to push images)
- This Week: 2.514 - started on time: You're invited to talk on Matrix
  - Built with JDK21 and Maven 3.9.10
Announcements:
- Security Advisory on the gatling plugin last Friday: Jenkins Security Advisory 2025-06-06
- AWS gave use addition 60k, we are really grateful!
- Next top level priority is to prepare getting away from the Azure sponsored subscription (ci.jenkins.io to AWS, then Redis instance, then ephemeral agents)
- Maven 3.9.10 released (tracked in Maven `3.9.10` upgrade campaign · Issue #4697 · jenkins-infra/helpdesk · GitHub)
  - 2.514 is the first release to be built using JDK21 and Maven 3.9.10

Upcoming Calendar

Next Weekly: 2025-06-17, 2.515
Next LTS: 2025-06-25, 2.504.3, Philipp Glanz is the release lead, ref. Events
- Not sure yet (no backport as for today)
Next Security Release as per jenkinsci-advisories: N.A.
- Surprise plugin advisory last week: Jenkins Security Advisory 2025-06-06
Upcoming credentials expirations (~3 weeks):
- 2025-06-14:
  - Azure SP used by infra.ci to spin up Azure VM agents: Azure AD Application password for Azure VM agents in `infra.ci.jenkins.io` expires on `2025-06-14T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1049 · jenkins-infra/azure · GitHub
    - Workload identity does not seem supported (while Managed Identity - VM) is (ref. Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials · Issue #4651 · jenkins-infra/helpdesk · GitHub) => we have to renew the credential
    - Issue: [infra.ci.jenkins.io] Azure Service Principal Credentials · Issue #4699 · jenkins-infra/helpdesk · GitHub
- 2025-06-15: Azure SP used to deploy websites to production in infra.ci (into file shares) - [infra.ci.jenkins.io] Azure Service Principal Credentials · Issue #4699 · jenkins-infra/helpdesk · GitHub
- 2025-06-16: (Issues to be done) Azure SP used for Terraform backends for projects (issue to create, with all other terraform state below):
  - AWS (CloudBees) - https://github.com/jenkins-infra/terraform-states/pull/56
- 2025-06-17: (Issues to be done) Azure SP used for Terraform backends (and some API tokens) for projects (issue to create):
  - AWS (Sponsored) - https://github.com/jenkins-infra/terraform-states/pull/57
  - Azure - https://github.com/jenkins-infra/terraform-states/pull/58
  - Azure Net - https://github.com/jenkins-infra/terraform-states/pull/59
  - Cloudflare - https://github.com/jenkins-infra/terraform-states/pull/60
    - @smerle to ensure @dduportal is not bus factor
  - Datadog - https://github.com/jenkins-infra/terraform-states/pull/61
  - Digital Ocean - https://github.com/jenkins-infra/terraform-states/pull/62
  - Fastly - https://github.com/jenkins-infra/terraform-states/pull/63
Next major event: N.A.

Cloud Budgets

Azure CDF:
- March: $4.3k (invoice)
- April: $3.9k (invoice)
- May: ~$3.2k (raw evaluation with CDF due to cost data hidden by Microsoft until billing issues are settled)
- June: $1,061 (forecast at ~$3.8k)
  - Slight increase on the forecast (DNS and bandwidth), no action needed until we’ve got rid of the Azure Sponsored subscription)
Azure Sponsorship (Microsoft Credits) - Remaining: $13,628 until 31 August 2025
- March: $4,276
- April: $12.1k
- May: $14.6k
- June: $2,980 => (forecast at ~$10k)
  - Slight decrease due to BOM improvement: You're invited to talk on Matrix
DigitalOcean - Remaining $13,772 until January 02, 2026
- March: $272 (invoice)
- April: $349 (invoice)
- May: $310 (invoice)
- June: $71 (forecast at $236)
AWS:
- CloudBees:
  - March: $551
  - April: $532
  - May: $548
  - June: $162 (forecast at $546)
- Sponsored account ($94,747 credits lefts until 2027)
  - March: $14,649
  - April: $1,977
  - May: $2,96
  - June: $0,87 (forecast at $6)
Jfrog Artifactory Usage
- Storage: 1.34TB (stable)
- Bandwidth:
  - March: 35.25 TB (better than expected)
  - April: 26.34 TB (22.25 Tb for ‘releases’ repository)
  - May: 23.93 Tb (22.21 Tb for ‘releases’ repository)
  - June: 6.5Tb (6.12 Tb for ‘releases’ repository, forecast at ~21 TB)
    - We have to provide Basil the access logs for studying

Notes

Done:
- Support:
  - Artifact caching proxy HTTP 403 fails many jobs on ci.jenkins.io
    - Fix deployed on ACP. The HTTP/403 came from Maven Central (not repo.jenkins-ci.org)
  - sg.mirror.servanamanaged.com mirror returns 404
    - They’re back, and their setup is up to date
Work in Progress:
- 2025 Cloud Usage: ensure that we can run until end of year
  - New set of sub issues (see triage)
- OSUOSL: decrease mirror dependency
  - [Documentation] add a public page with the “add a jenkins mirror” procedure
    - New source of truth on Mirrors
    - New helpdesk issue template for mirror
    - WiP: document the “1 year of artifacts”
  - Increase in mirror size post migration to archives.jenkins.io
    - Blocked by documentation missing
  - Then: apply archives.jenkins.io Ip restrictions + deploy new set of scripts
- Support:
  - [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
    - Damien meets Andrew later today for March statistics
    - Attention point: let’s see if we can generate month partial data + avoid holes in data set
  - Build cache for jenkinsci/ath
    - Wip: @jay started to implement the support of cache-to, draft PR for initial review
  - jgit in Artifactory proxies all of Eclipse and jgit-cache contains 200GB
    - Wip: Darin is adding patterns to only cache what we need
    - No cache cleanup (yet)
  - Restore jenkinsci/jenkins build stability
    - Let’s remove it from milestones as no actions from the Jenkins Infra
- Keep infrastructure up to date:
  - Update Jira LTS from 9.12.x to 10.3.x
    - LF is building a new VM as new JDK is required (and new OS)
    - Most probably in July
    - LTS is EOL in November 2025
  - Use JDK21 Platform-wide
    - Switch agent (java home/Path) to JDK21 default
      - Docker Packaging done
      - WiP on the ci.jenkins.io agent (PR opened to be merged), and eventuall BOM
  - JDK25 integration
    - After JDK21 and cloud migration, but we can start some work on it in the meantime
- Keep infrastructure sane and sustainable:
  - Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
    - @jay is blocked by the “report” publication destination. reports.jenkins.io is not a valid location: we need a new webservice for this (safety measure).
  - Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials
    - Does not seem supported by the Azure Credential plugins (unlike VM Managed Identity). Back to backlog until it is supported.
  - [publick8s/datadog] Audit datadog logs collection
    - Nothing done, low priority, but easy one.
Issues staying in backlog/triage:
Issues added to the next milestone:
- Maven 3.9.10 upgrade campaign
  - @jay needs more right on packer-image
  - Docker packaging done
  - WiP: packer-image to be fixed by bumping Maven
  - Then, find the technique used on docker-packaging for not failing on old maven version (3.9.9) so the problem does not happen again
- Create a “private report” web service for monitoring
  - Need to specify task list in details
- [infra.ci.jenkins.io] migrate container agents (cluster infracijenkinsio-agents-1) from Azure (sponsored) subscription to Azure (CDF) subscription
- [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account
- Make sure typos checks run on ci.jenkins.io
- Failed publishing GH check: Resource not accessible by integration
- [jira-plugin] Crowd setup

Topic	Replies	Views
Infrastructure Team Meeting - June 17, 2025 Infrastructure meeting , sig-infra	13	June 19, 2025
Infrastructure Team Meeting - June 24, 2025 Infrastructure meeting , sig-infra	9	June 26, 2025
Infrastructure Team Meeting - May 06, 2025 Infrastructure meeting , sig-infra	17	May 10, 2025
Infrastructure Team Meeting - February 25, 2025 Infrastructure meeting , sig-infra	19	February 25, 2025
Infrastructure Team Meeting - June 03, 2025 Infrastructure meeting , sig-infra	21	June 7, 2025

Infrastructure Team Meeting - June 10, 2025

Attendees

Announcements

Upcoming Calendar

Cloud Budgets

Notes

Related topics