Infrastructure Team Meeting - May 13, 2025

[dduportal (Damien Duportal)]

2025 05 13 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.509 was susccessful: You're invited to talk on Matrix
   • This week: 2.510, started on time, we had a credential issue but it is now fixed. Expect a 1h30 delay (comapred to usual) - You're invited to talk on Matrix
2. Announcements:
   • OSUOSL secured their funding of $250k for this year: Forging Our Future: OSL's Path to Sustainability – A Call for Smart Solutions and Enduring Support | OSU Open Source Lab
   • Microsoft won’t give us more credits BUT they accepted to extend the expiration date of current credit to end of August (+ 3 months)

Upcoming Calendar

• Next Weekly: 2025-05-20, 2.511
• Next LTS: 2025-05-28, 2.504.2, Kris Stern is release lead
• Next Security Release as per jenkinsci-advisories:
  • 2025-05-14: plugins advisory - https://groups.google.com/g/jenkinsci-advisories/c/AivXXJyz650/m/-m4yFXHWAAAJ
• Upcoming credentials expirations (~3 weeks):
  • 2025-05-22:
    • (Issue to be done) Azure SP used by updatecli in infra.ci: Extend Azure AD Application password validity for packer-resources (current end date: 2025-05-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #1030 · jenkins-infra/azure · GitHub
• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • February: $3.9k (invoice)
  • March: $4,3k (invoice)
  • April: $3,9k (invoice)
  • May: $1,381 (forecast at $3.2k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $26,408 until 31 August 2025

  • February: $11.2k
  • March: $4,276
  • April: $12.1k
  • May: $4,877 (forecast at $12k)

• DigitalOcean - Remaining $14,300 until January 02, 2026

  • February: $237 (invoice)
  • March: $272 (invoice)
  • April: $349 (invoice)
  • May: $147 (forecast at $379)

• AWS:

  • CloudBees:

    • February: $550
    • March: $551
    • April: $532
    • May: $203 (forecast at $536)

  • Sponsored account (~$36,727 credits lefts until 01/31/2027)

    • February: $8.5k
    • March: $14,649
    • April: $1,977
    • May: $1,12 (forecast at $6)

• Jfrog Artifactory Usage

  • Storage: 1.3xTB
    • jcenter cleaned up
  • Bandwidth:
    • March: 35.25 TB (better than expected)
    • April: 26.34 TB (good! Mostly ‘releases’ repo: 22.25Tb, then 1.21 Tb on jcenter and 1 Tb on incrementals)
    • May: 7,89 Tb (7,4 Tb is ‘releases’, forecast at ~20 TB)

Notes

• Done:

  • Keep the platform up to date/in good shape:
    • [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) expires the 2025-05-14
      * We should enhance the issue and/or PR to detail how to force-apply the new credential value + add a test job => @smerle
    • [infra.ci.jenkins.io] 2025-05-15 Expiration of the Digital Oceans PATs used by Terraform
    • Tombstone OSUOSL VMs edamame and lettuce (and leftovers of radish and okra)
    • Migration left over from publicK8s to arm64
    • Certbot: keep it up to date (along with its plugins)
      • To reopen as the puppet rspec tests need also to be updated by the updatecli manifest

• Work in Progress:

  • Keep the platform up to date/in good shape:
    • [docker-builder] bump to arm64 compatible
      • Required before using packer all in one instead (which requires arm64)
      • Thanks Gavin for providing an arm64 version of typo-checkstyle tool (need to bump version to 0.2.0)
      • typo was also required to be tracked and bumped
      • WiP on the aarch64/arm64 compatibility of the tools in the image: almost there
    • OSUOSL: decrease mirror dependency
      • Knowledge Sharing session around all the scripts involved in this. Need to provide diagrams with the upcoming
      • Next step: work on archives as reference mirror
    • Update Jira LTS from 9.12.x to 10.3.x
      • No news from the Linux Foundation
    • packer-images: use api.adoptium.net for updatecli and installation of JDKs
      • Preliminary for upcoming JDK25 tool for developers
      • Exploratory work done, currently splitting in PR per feature starting with the current updatecli manifests
    • [puppet.jenkins.io] Migrate to DigitalOcean
      • Failed to use a Docker image for puppet controller using official images
        • Cannot use pupperware Docker Compose stack and cannot restore backup from PE to Puppet OSS
      • Need another exploratory session (container as VM technique) or close it and focus on Ansible instead
      • Let’s remove current DO resources for this as we do not use them (and keep credits) => @smerle
  • 2025 Cloud Usage: ensure that we can run until end of year
    • Need to add issue for ci.jio BOM and then delaying of 1 week
  • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
    • Damien meets with Andrew Thursday, then Knowledge sharing with Stephane
  • Use JDK21 Platform-wide
    • Delayed to next week
    • Move controllers to JDK21 (runtime)
    • Move agents to JDK21 (runtime)
  • Ubuntu 22.04 upgrade campaign
    • Back to backlog until we solve the puppet problem + migrate pkg.origin to Kubernetes

• New Issues (to triage):

  • JDK25 integration
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools

• ToDo (next milestone) (GitHub · Where software is built)