Infrastructure Team Meeting - April 21, 2026

Contributing Infrastructure
,
1

Attendees :busts_in_silhouette:

Announcements :loudspeaker:

  1. Jenkins Weekly Releases
    • Last Week: 2.559
      • Infrastructure issue, delayed the image of 2 hours (packer image)
    • This Week: 2.560
      • Everything went fine
  2. Announcements:
    • Jenkins Digicert Code Signing Certificate expires in May: [pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026 · Issue #4923 · jenkins-infra/helpdesk · GitHub
      • Mark successfully used LF’s signing system locally
      • Started to implement in our release process.
        • Short term: using Azure credential provided by LF.
          • WiP on Windows agent environment (tools: dotnet 8, etc.) dedicated for signing only (slower build, faster to test and implement)
        • Medium term: move Windows environment (build AND signing) to our all-in-one VM (faster build, less maintenance)
        • Long term: add OIDC on release.ci with a public issuer endpoint to allow credential-less signing with LF (safer for them)
      • Proposed calendar:
        • Targeting new signing for weekly 2.561 on 28 April
        • Then 2.552.2 LTS on 13 May
    • Incoming updates:
      • (Critical) Incoming Java patch campaign
      • (Important) Node JS 24.x
        • packer image is on 22, need to be bumped
        • GHA are deprecating NodeJS 20 actions (security-scan, CD releases, and helpdesks)
      • (Important) Docker CE 29+
        • Should fix Windows issues
        • If it breaks plugins CI builds, then they must bump their docker-client API version
        • ATH is already using 29.x in their nested Docker engines
      • (Nice to have) Maven 3.9.15
        • 3.9.14 was delayed. let’s jump directly to 3.9.15.
      • (Nice to have) Helm v4
    • Team capacity:
      • Herve is off this Friday and next week
    • Priorities:

Upcoming Calendar :tear_off_calendar:

  • Next infra meeting: 2026-04-28
  • Next Weekly: 2026-04-28 - 2.561
    • Targeting new MSI signing
    • Hoping release-drafter fixed (as per Herve’s feedback in Elements jenkinsci/release)
  • Next LTS: 2026-05-13 - 2.555.2 - Release Lead: Mark Waite
    • RC: 2026-04-29
    • Targeting new MSI signing
    • Note: Infra team to update the release checklist as per their feedbacks. 2 focus points: timing and release.ci build (parsing/failing/parent/child)
  • Next Security Release as per jenkinsci-advisories: N/A
  • Upcoming credentials expirations (~3 weeks):
    • 2026-05-01: Azure Credential used by Packer Image builds (issue to be created) => Damien + Jay
    • 2026-05-13: NPM token used by infra.ci.jenkins.io (issue to be created)
    • 2026-05-14: Netlify token used by infra.ci.jenkins.io (issue to be created)
    • 2026-05-15: Azure File Share Credentials for (contributors|stats|plugins).jenkins.io websites used by infra.ci.jenkins.io (issue to be created)
  • Next major event:
    • cdCon May 18-20 in Minesota, Mark will be controller of ceremonies!

Cloud Budgets

  • Azure (CDF paid)

    • February: $6.4k (invoice)
    • March: $5.6k (invoice)
    • April: $2.5k (forecasted at $3.7k)
      • Cost reduction effort is paying
      • Additional cost reductions are still ongoing

  • Azure Sponsored subscription (Microsoft Credits): $94,534 credits left

    • March: $1847 (invoice)
    • April: $3.6k (forecasted at $5.2k)
    • TODO: expiration date? (Damien)

  • DigitalOcean - Remaining $8,761.11 until January 02, 2027 (~11 months left at current rate)

    • February: $565.03 (invoice)
    • March: $423.63 (invoice)
    • April: $558.89 (forecasted at $798)

  • AWS:

    • Sponsored account, credits left: $13,880.67 until 05/31/2027, and $60k until 01/31/2028
      • February: $11.1k (invoice)
      • March: $14.0k (invoice)
      • April: $9.8k (forecasted at $15.0k)
        • Still high (EC2 plugin bug, BOM, non spots, old instance family, bandwidth)

  • JFrog Artifactory Usage

    • Storage: 1.83 Tb
      • Slight increase, no need to check (too small)
    • Bandwidth:
      • February: 17.46 Tb
      • March: 27.24 Tb
      • April: 30.23 Tb (forecasted at ~44 Tb)
        • Mark/Damien needs to warn JFrog about the heavy use in March and April
        • Mark needs to review the recent log analysis

Notes :book:

1 Like