Infrastructure Team Meeting - August 05, 2025

[dduportal (Damien Duportal)]

2025 08 05 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.521 - success - You're invited to talk on Matrix
     • DockerHub issues (typical Tuesday) - some layers of some images reported internal error, try again later
   • This Week: 2.522 - started on time - You're invited to talk on Matrix
     • Using a new cluster, watch packaging very carefully
2. Announcements:
   • Stephane is off until end of August
   • Infra Roadmap
     • Jenkins Roadmap
     • Priority: finish Azure Sponsored subscription decommission
       • Redis database migration and its consumers

Upcoming Calendar

• Next Weekly: 2025-08-12 - 2.523
• Next LTS: 2025-08-20 - 2.516.2
  • RC: 2025-08-06 (tomorrow)
• Next LTS baseline will be selected 2025-09-03
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-08-18: [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expires the 08 August 2025
  • 2025-08-31: [infra.ci.jenkins.io] Netlify token expires the 31 August 2025

Cloud Budgets

• Azure CDF - Remaining: $32,154 for 2025:

  • May: $3.5k (invoice)
  • June: $3,474 (invoice)
  • July: $4,289 (expecting invoice at ~4.4k)
  • August: $755 (forecast at ~6.5k , we can only afford 6.4k monthly until end of 2025)
    • Increase due to workload migrated last week (ephemeral VM agents for trusted,cert and infra, packer builds)
    • Need to quickly analyse and act

• Azure Sponsorship (Microsoft Credits) - Remaining: $905 until 31 August 2025

  • May: $14.6k
  • June: $10.5k
  • July: $5,1k
  • August: $125
    • ~31$ daily in the past 5 days => 29 days left at this rate

• DigitalOcean - Remaining $13,343.55 until January 02, 2026

  • May: $310 (invoice)
  • June: $252 (invoice)
  • July: $196 (invoice)
  • August: $25 (forecast at $194)

• AWS:

  • CloudBees:

    • May: $548
    • June: $606
    • July: $635,75
    • August: $99.40 (forecast at $616)

  • Sponsored account ($89,470 credits lefts until 2027 => ~17 months remaining)

    • May: $2.96
    • June: $367.47
    • July: $4,899.99
    • August: $748 (forecast at $5.2k)

• Jfrog Artifactory Usage

  • Storage: 1.24TB (slight increase of 10 Gb due to logs)
  • Bandwidth:
    • May: 23.93 TB (22.21 Tb for ‘releases’ repository)
    • June: 22.94 TB (21.44 Tb for ‘releases’ repository)
    • July: 24.96 Tb (23.45 Tb for ‘releases’ repository)
    • August: 3.6 Tb - (3.21 Tb for ‘releases’ repository) - forecast at 23 Tb

Notes

• Done:

  • Decommission Azure Sponsored Subscription:
    • [privatek8s] migrate cluster privatek8s from Azure (sponsored) subscription to Azure (CDF) subscription
  • Support
    • Install theme plugins on ci.jenkins.io
      • Updated the plugins as well
    • Expired SSL certificate on issues.jenkins.io
      • No alert from Datadog?
      • Assumed they were using certbot, but a new technique is being used instead
      • Changed from LetsEncrypt to AWS for SSL certificate creation
        • Mark create a helpdesk ticket to monitor SSL certificate expiration with Datadog
    • Migrate buildtrigger-plugin wiki to github
    • Maven cache on ci.jenkins.io is more than 7 days old
      • Increase resource limits, write archive directly to AWS S3 bucket as a file system (good write performance)
        • Extends 9 minute build by 15 seconds
    • docker-ssh-agent builds consistently timeout on ci.jenkins.io
      • Outdated SSH.exe on the Windows images removed
  • Keep platform up to date:
    • JDK patch upgrade campaign (July 2025)
    • [private.vpn.jenkins.io] 2025-08-08 (August 2025) VPN CRL expires
    • [release.ci.jenkins.io] Azure Credentials for Core Release (Vault access) expires the 2025-08-10
    • [infra.ci.jenkins.io] 2025-08-08 Expiration of the Digital Oceans PATs used by Terraform
  • Keep platform sane and maintainable:
    • [Azure] Decommission the prodreleasecore Storage Account

• Work in Progress:

  • 2025 Cloud Usage: ensure that we can run until end of year
    • Decommission Azure Sponsored Subscription
  • Support:
    • Admin access for Jenkins GSoC org admins to GSoC SiG Gitter channel
      • No answer from Gitter: @dduportal to ping them again
    • Github Copilot organisation space
      • Nothing happenned from GitHub
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • First slice of knowledge with Stephane (usage.jio migration to DO to allow generating stats manually once - need pgsql and a lot of disk)
    • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
      • Forgot an attribute in the migration, searched history and found it, missed in the cherry pick
      • Attribute is now updated, need to wait 5-7 days to finally confirm the garbage collection is working as expected
        • Move the garbage collection job to infra.ci
        • Must scale the service down, remove the files, scale the service up
  • Keep platform up to date:
    • Update Jira LTS from 9.12.x to 10.3.x
      • Damien is late: need to create a new LDAP binding user to unblock Ryan at LF
  • Keep platform sane and maintainable:
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
      • Pipeline Library has been published and can be used (verified with docker-404 pipeline). Improved with the azcopy logs if it fails (to ease diagnostic when credential or network issues)
      • WiP: Datadog synthetics monitors
        • Issue commented with the 3 “initial” conditions which failure would trigger in alert, scoped to only docker-404 pipeline/project
    • Reduce artifactory bandwidth used by ci.jenkins.io
      • ACP was not caching some files (war files). Configuration mistake, now fixed. ACP is now caching way more things!
      • Could use the Maven cache from BOM in other builds
    • Reduce artifactory bandwidth used by infra.ci.jenkins.io
      • No work done, shared new IP addresses with Darin Pope
    • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
      • Delayed
    • Tombstone Puppet (and replace it by something else)
      • Delayed
    • OSUOSL: decrease mirror dependency
      • Delayed, until Update center is HTTPS enforced
    • Deprecate ircbot service
      • Not clear which services are still in use
      • Docs SIG needs a change on the public documentation, needs review by others

• Issues staying in backlog/triage:

  • [infra.ci.jenkins.io] Netlify token expires the 31 August 2025
  • private docker image registry for staging core security releases
  • [Azure]: credentials less Service Principal
  • Tracking Issue for Groovy Script Conversion in RPU
  • Artifactory Not Caching Google’s Maven Repository
  • Migrate census.jenkins.io VM from AWS CloudBees to DigitalOcean
  • [get.jenkins.io/updates.jenkins.io] Move the managed redis-public instance out of Azure (sponsored) subscriptions
  • Chinese jenkins site incorrect site redirection
  • [Azure] Migrate (e.g. re-create) AKS clusters publick8s and privatek8s with modern settings (private API, Azure Linux, NAT outbound)
  • [cert.ci/trusted.ci/private.vpn] Default outbound access for VMs in Azure will be retired
  • [private.vpn.jenkins.io] Azure deprecates Public IPs of type “Basic” the 30 September 2025
  • Add a real-world job to weekly.ci.jenkins.io
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Add monitoring for CD secrets updates
  • [INFRA-3046] Monitor Jenkins mirrors Age

• Issues added to the next milestone:

  • [Update Center] Enforce HTTPS for updates.jenkins.io and updates.jenkins-ci.org
  • Mirror mirrors.de.sahilister.net - Network is unreachable
  • Inconsistent labels result in missing JDK 21 installers
  • [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expires the 08 August 2025
  • GSoC 2025 - Plugin Modernizer improvement - New repo for metadata-plugin-modernizer