Infrastructure Team Meeting - July 21, 2026

dduportal · July 21, 2026, 11:56am

Meeting cancelled. Notes/Milestone will be published though but Damien was unable to attend and prepare.

Announcements

Jenkins Weekly Releases
- 2 weeks ago: 2.572
  - Issue on the Docker Windows 2025 (hcs::CreateComputeSystem). Had to replay with Herve’s pipeline change (sequentialize JDK21 and JDK25).
- Last Week: 2.573
  - Same issue on the Docker Windows 2025 (hcs::CreateComputeSystem). Had to replay with Herve’s pipeline change (sequentialize JDK21 and JDK25).
- This Week: 2.574
  - No issue! The Docker “sequential” build on Windows works well. Thanks Herve!
Announcements:
- Meeting cancelled. Notes/Milestone will be published though but Damien was unable to attend and prepare.
- Team capacity:
  - Herve is off until Monday August 3rd with limited availability on 3/4/5 Aug.
  - Jay is not available until Thursday 24 July
- Priorities:
  - Infra Roadmap: Jenkins Roadmap
  - Current EPICs (by priority): GitHub · Where software is built

Upcoming Calendar

Next infra meeting: 2026-07-28 (next week)
Next Weekly: 2.574 - 2026-07-28 (next week)
Next LTS: 2.568.2 - 2026-08-05
- RC tomorrow (Wed. 2026-07-22)
Next Security Release as per jenkinsci-advisories: N.A.
Upcoming credentials expirations (~3 weeks):
- 2026-07-26: Azure Credential used by Packer Image builds expires
  - Issue to created by @dduportal
- 2026-07-29: Algolia API write key expires (plugins.jenkins.io)
  - Issue to created by @dduportal
- 2026-08-03: Netlify token used on infra.ci to deploy websites (preview/CD) expires
  - Issue to created by @dduportal
- 2026-08-05: [infra.ci] Azure Credential to deploy docs.jenkins.io expires
  - Issue to created by @dduportal
- 2026-08-09: [infra.ci] contributors|stats|plugins FileShare service principal credentials
  - Issue to created by @dduportal
Next major event: N.A.

Cloud Budgets

Azure (CDF paid)
- April: ~$3,553 (invoice)
- May: ~$2,800 (invoice)
- June: ~$2,108 (invoice)
- July: ~$1,445 (forecasted at $2,2k)
  - We will meet our threshold of $2.5k monthly
  - Note: publick8s should move to sponsored subscription ($880 for June, forecasted at $960 for July)
Azure Sponsored subscription (Microsoft Credits): $80967 credits on July 1st 2026 - until 20 Oct. 2027
- April: ~$5,902 (invoice)
- May: ~$5,734 (invoice)
- June: ~$5,542 (invoice)
- July: ~$3221 (forecasted at $5.0k)
  - At current rate, we shall deplete credits on October 2027 (unless we have to move ci.jenkins.io here)
DigitalOcean - Remaining $6,472.29 until January 02, 2027
- April: $841,69 (invoice)
- May: $1186.97 (invoice)
- June: $740.32 (invoince)
- July: $321.40 (forecasted at ~$498)
  - At current rate, we shall deplete credits on February 2027
AWS:
- Sponsored account, credits left: $35.7k until 01/31/2028 (on 1st July)
  - April: $15.4k
  - May: $14.9k
  - June: ~$7,893
  - July: ~$3,740 (forecasted at ~5.6k)
    - -29% compared to June
    - At current rate ($5.6k monthly), we shall deplete credits on January 2027
JFrog Artifactory Usage
- Storage: 1.87 Tb (steady)
- Bandwidth:
  - April: 43.85 Tb
  - May: 29.98 Tb
  - June: 23.24 Tb
  - July: 18.86 TB (forecasted at ~28 Tb)
Algolia API usage
- Context: [plugins.jenkins.io] Watch Algolia search usage for June 2026
- plugins.jenkins.io:
  - (July): ~16.6K (16%) search requests(max. 400K monthly)

Notes

Done:
- keep-infra-sane-and-maintainable
  - Identify any existing uses of potentially unused features of the account app
    - Thanks Daniel Beck! It follows up the update of accounts.jenkins.io for a few security issues.
  - Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.568.1
  - Nobody noticed that the last successful extension-indexer run was in August 2023
    - Was already implemetend in @7JoquqrvQFu6FDlR14JVhg 's past setups in May/June
- support
  - Add nevingeorgesunny to jenkinsci GitHub organization
    - Thanks Tim!
  - NPM 12.x breaks our Gastby-based websites using sharp
    - Had to override the sharp dependency to fix this problem.
    - pin (and track) NPM version · Issue #2921 · jenkins-infra/packer-images · GitHub for long term
  - repo.jenkins-ci.org currently returning 403 error
    - Jfrog’s issue due to CNAME misconfigured since years (we did not know).
    - all fixed!
  - checks stuck on jenkins-infra/stories
    - AWS capacity issue with arm64. Fixed by using other kinds.
    - Long term: [pipeline-library] define a common `nodeWithTimeoutAndRetry()` function for website builds · Issue #5229 · jenkins-infra/helpdesk · GitHub
  - Docker plugin release failed from CD with HTTP 401
    - Consequence of the Artifactory (repo.jenkins-ci.org) outage.
  - Low available disk space on multiple jenkins-io PVCs (publick8s + privatek8s-sponsored)
    - We did not have to increase the NFS volume size: cleaning up staging was good enough. Might not be the case in the future.
  - Low available inodes on multiple jenkins-io PVCs (publick8s + privatek8s-sponsored)
    - Closed with previous one after clean up.
  - CD release of finite-state-analysis-plugin fails with 401 deploying to repo.jenkins-ci.org
    - Running RPU every 2h instead of 3h was good enough.
    - Nice to have: update our monitoring threshold for RPU => @dduportal
- keep-infra-up-to-date
  - Prepare release, packaging and docker backports for LTS 2.568.1
  - [infra.ci.jenkins.io] DigitalOcean PAT used by terraform expires on the 2026-07-19
    - Improvement point: take time for Sops → Kubernetes → mounted secrets → JCasC reload → Job multibranch scan to happen.
    - Improvement point: add @7JoquqrvQFu6FDlR14JVhg as infra.ci admin to allow JCasC reload
  - Bump Helm from 3.x to 4.x
    - Impacted (silently) our helm-charts GHA.
    - Long term: move helm GHA to infra.ci.
    - Nice to have: report issue to helmfile for the deprecated --atomic flag => @7JoquqrvQFu6FDlR14JVhg
  - Upgrade jq to 1.8 on packer-images and ensure compatibility across all jenkins-infra projects
    - N.A.
Closed as not planned:
- I am trying to gain access or register
- [INFRA-1560] Investigate setting Expect-CT header for jenkins.io domains
Work in Progress:
- support
  - CD release fails with 403 Forbidden deploying testingbot plugin to repo.jenkins-ci.org/releases
    - Looks like an issue in RPU which does not associate a group withb permission scheme (but associate user instead).
  - Infra build failing for plugin-site
    - Long term: [pipeline-library] define a common `nodeWithTimeoutAndRetry()` function for website builds · Issue #5229 · jenkins-infra/helpdesk · GitHub
    - But we also need to implement a retry in the netlify publication command
  - build-pipeline plugin tests now fail with ‘cannot find Chrome binary’
    - Caused by removal of the (ancient) Chrome installation from agent templates.
    - Replaced by playwright.
    - We’ve added a symlink to /usr/local/bin/chromium to ensure it can still be used.
    - Still open: the plugin need to update its Selenium setup to support Chrome 149 (stuck on older version)
- keep-infra-sane-and-maintainable
  - Migrate publick8s AKS cluster to the Azure 2026 Sponsored subscription
    - Need to write it down => @dduportal
    - Important note: worth exploring used Sweden instead of US East 2 (due to Azure capacity issues in the US)
  - Decrease BOM costs by splitting tests more efficiently
    - WiP by Herve. Promising results on his initial PoC implementation: for a 1h build, max. 40 agents (20 if only weekly line)
    - On hold (back to backlog) until he is back from holidays.
  - [AWS] Activate user-defined cost allocation tags
    - We’ve added a few more tags. We start to see them but still not as useful as expected to track specific costs.
    - Need more tags a bit more tag to start significant exploration
- keep-infra-up-to-date
  - Prepare update-center root CA rotation
    - @dduportal back from holidays: will start working on it
- Drop Windows 2019 support
  - Controller container image dropped the support
  - Next step: agent container images (need to open draft PRs as support for discussion)
Issues added to the next milestone:
Issues staying in backlog/triage:

Topic	Replies	Views
Infrastructure Team Meeting - June 23, 2026 Infrastructure meeting , sig-infra	39	June 25, 2026
Infrastructure Team Meeting - August 04, 2026 Infrastructure meeting , sig-infra	38	August 5, 2026
Infrastructure Team Meeting - June 2, 2026 Infrastructure meeting , sig-infra	34	June 5, 2026
Infrastructure Team Meeting - July 28, 2026 Infrastructure meeting , sig-infra	33	July 29, 2026
Infrastructure Team Meeting - July 07, 2026 Infrastructure meeting , sig-infra	34	July 17, 2026

Infrastructure Team Meeting - July 21, 2026

Announcements

Upcoming Calendar

Cloud Budgets

Notes

Related topics