Infrastructure Team Meeting - June 17, 2025

[dduportal (Damien Duportal)]

2025 06 17 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.514 - successful release: You're invited to talk on Matrix
     • Built with JDK21 and Maven 3.9.10
   • This Week: 2.515 - started on time: You're invited to talk on Matrix
2. Announcements:
   • Git 2.50.0 (Windows long path \o/)
   • Docker Windows failures on ci.jenkins.io since a few days: let’s watch weekly release carefully
   • New contributor to the infra: fix(jenkinscontroller): Use non-deprecated inbound agent URL arguments by zhan9san · Pull Request #4188 · jenkins-infra/jenkins-infra · GitHub
   • jenkinsistheway.io domain is not ours: Should we try to rebuy it?
     • If no, then we have to cleanup a bit: removing DNS zone + records, decomission the Kubernetes redirector service (and the httpredirector chart)
     • General Board task: CDF should own the Jenkins domains
       • jenkinsistheway.io => it’s been a long time. Let’s decommission! (search source code to update links) => @dduportal create issue for decommission
       • Action item for domains jenkins.io and jenkins-ci.org which are owned by Tyler, to transfer it to CDF/LF => @en3hD3iMRx6_6IXLNY0Rag

Upcoming Calendar

• Next Weekly: 2025-06-24: 2.516
• Next LTS: 2025-06-25, 2.504.3, Philipp Glanz is the release lead, ref. Events
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-07-06:
    • (Issue to create)- Azure SP used for updatecli jobs in infra.ci: Azure AD Application password for updatecli in `infra.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1063 · jenkins-infra/azure · GitHub
    • (Issue(s) to create)- Azure SP used to deploy websites to production from trusted.ci (into file shares):
      • www.jenkins.io: Azure File Share Principal `www.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1062 · jenkins-infra/azure · GitHub
      • javadoc.jenkins.io: Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1061 · jenkins-infra/azure · GitHub
      • good candidate for moving to a credential-less by using ephemeral VM agent assigned identity (trusted.ci is NOT multi tenant)
    • (Issue to create)- Azure SP used to deploy websites to production from infra.ci (into file shares):
      • docs.jenkins.io: New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2025-07-06T00:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #1059 · jenkins-infra/azure · GitHub
    • (Issue to create)- Azure SP used by geoip Jobs in cluster publick8s to update geolocation DB for mirrorbits: Azure File Share Principal `geoip` on `publick8s` expires on `2025-07-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #1060 · jenkins-infra/azure · GitHub
      • good candidate for moving to a credential-less workload identity. Jenkins Azure plugin do not support it (yet) as per Jenkins Controllers in Azure Kubernetes: use workload identity management to allow managing Azure VM agents without credentials · Issue #4651 · jenkins-infra/helpdesk · GitHub BUT this one is a direct pod → Azure File Storage thing so it should work well.

Cloud Budgets

• Azure CDF:

  • March: $4.3k (invoice)
  • April: $3.9k (invoice)
  • May: ~$3.5k (invoice)
  • June: $1,844 (forecast at ~$3.7k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $10,946 until 31 August 2025

  • March: $4,276
  • April: $12.1k
  • May: $14.6k
  • June: $5671 => (forecast at ~$10k)
    • Slight thanks due to BOM improvement
    • We shall have ~5k to consume after ci.jio migration (for the other services) => no rush for the migration (can be 1st week of July)

• DigitalOcean - Remaining $13,685.03 until January 02, 2026

  • March: $272 (invoice)
  • April: $349 (invoice)
  • May: $310 (invoice)
  • June: $159 (forecast at $279)

• AWS:

  • CloudBees:

    • March: $551
    • April: $532
    • May: $548
    • June: $296 (forecast at $550)

  • Sponsored account ($94,747 credits lefts until 2027)

    • March: $14,649
    • April: $1,977
    • May: $2,96
    • June: $76,31 (forecast at $282)
      • We’ve started creating resources to support ci.jio migration in ~2 weeks (hence the slight increase)

• Jfrog Artifactory Usage

  • Storage: 1.34TB (stable)
  • Bandwidth:
    • March: 35.25 TB (better than expected)
    • April: 26.34 TB (22.25 Tb for ‘releases’ repository)
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 11.79TB (11.09TB Tb for ‘releases’ repository, forecast at ~20.8 TB)

Notes

• Done:

  • Support:
    • Build cache for jenkinsci/ath
      • Our pipeline library used to build/deploy Docker images now supports “cache-to” (exporting build cache using buildx engine to a registry or anywhere else)
    • jdk25 new version failed our updatecli checks
      • Temurin API is not really stable for EA (or their release process)
      • We disabled tracking EA new versions for now to avoid failures
      • Issue opened on Temurin tracker
    • Unable to login with newly-created jenkinsio account (trying to file a bug)
    • 403 when wget jenkins.war
    • Failed publishing GH check: Resource not accessible by integration
  • Keep platform sane and maintainable:
    • [Terraform Backends] Expiration of credentials for Backend States and Cloudflare API tokens the 2025-06-17
    • [trusted.ci.jenkins.io] Update Center credential for Cloudflare R2 expires on 2025-06-17
    • [infra.ci.jenkins.io] Azure Service Principal Credentials
    • [Documentation] add a public page with the “add a jenkins mirror” procedure
  • Keep platform up to date:
    • Switch agent (java home/Path) to JDK21 default
    • Use JDK21 Platform-wide
    • Maven 3.9.10 upgrade campaign

• Work in Progress:

  • 2025 Cloud Usage: ensure that we can run until end of year

    • [infra.ci.jenkins.io] migrate container agents (cluster infracijenkinsio-agents-1) from Azure (sponsored) subscription to Azure (CDF) subscription
      • No blockers, vnet done, cluster terraform definition in progress (first peer review done today)
    • [ci.jenkins.io] migrate from Azure (sponsored) subscription to AWS (sponsored) account
      • No blockers, vnet done, S3, controller VM and “endpoints” done

  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)

    • WiP on the pipeline library (with a shell script for report generation). Unit tests now.
    • Blocker now: Create a “private report” web service for monitoring
      • reports.jenkins.io holds data which should not be available to “every job” on infra.ci or other controllers

  • Keep platform sane and maintainable:

    • OSUOSL: decrease mirror dependency
      • Increase in mirror size post migration to archives.jenkins.io
    • [publick8s/datadog] Audit datadog logs collection
    • jgit in Artifactory proxies all of Eclipse and jgit-cache contains 200GB
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Step 1: Load data from gzipped logs to pgsql database
      • Step 2: Export data from pg to github repo (and then regenerate the stats.jenkins.io)

  • Keep platform up to date:

    • Update Jira LTS from 9.12.x to 10.3.x

  • Support:

    • Make sure typos checks run on ci.jenkins.io
      • Only typos is required now on all in one. typos-checkstyle can be forgotten. Thanks @zbynek!
    • [jira-plugin] Crowd setup
    • JDK25 integration
      • All in one with JDK25+26 is available on ci.jio
      • Next tools: tools (and jenkins infra doc)

• Issues staying in backlog/triage:

  • [infra.ci.jenkins.io] Move packer-images workload in Azure from the sponsored subscription to the CDF subscription
  • Artifactory Not Caching Google’s Maven Repository
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • [trusted.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [cert.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [infra.ci.jenkins.io] migrate VM ephemeral agents from Azure (sponsored) subscription to Azure (CDF) subscription
  • [privatek8s] migrate cluster privatek8s from Azure (sponsored) subscription to Azure (CDF) subscription
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates

• Issues added to the next milestone:

  • javadoc.jenkins.io corrupted rendering due to missing stylesheet
    • We have to move this to javadoc repository
  • [ci.jenkins.io] Docker (*) images builds are all failing for Windows