Infrastructure Team Meeting - August 18, 2026

[dduportal (Damien Duportal)]

2026 08 11 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @lemeurherve (Hervé Le Meur)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)

Announcements

1. Jenkins Weekly Releases

   • Last Week: 2.577 (minor: fastly cache purge failed again)
     • Fastly cache purge helpdesk issue to open => @dduportal
   • This Week: 2.578 (In progress, started a bit later than usual due to a stuck pipeline from 2.577)
     • Herve following the release with the help of Mark

2. Announcements:

   • Team capacity:
     • Hervé is off from 19 Aug. 2026 (tomorrow) and back on 31 Aug.
     • Damien is off from 19 Aug. 2026 (tomorrow) and back on 24 Aug.
       • Also off on the 31 Aug.
     • Jay is off on 31 Aug. and 1st Sep.
   • Priorities:
     • Infra Roadmap: Jenkins Roadmap
     • Current EPICs (by priority): GitHub · Where software is built
     • Topic worth mentioning (timeframe of the upcoming 2 months)
       • BOM on cert.ci
       • mirrorbits v0.6.2 (from 0.6.1)
       • Nginx ingress retirement
       • Kubernetes 1.35 upgrade
     • Sponsoring: NodeOPS

Upcoming Calendar

• Next infra meeting: 2026-08-25
• Next Weekly: 2026-08-25 - 2.579
• Next LTS: 2026-09-02 - 2.568.3
  • Next baseline selection: 2026-08-19 (tomorrow)
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2026-08-26:
    • [infra.ci] Fastly Automation token used by Terraform expires => issue to create @duportal
    • [infra.ci] Fastly Automation token used to purge (plugins|contributors|etc.) @duportal
  • 2026-08-27:
    • [trusted.ci] Renew Cloudflare R2 Bucket Credential => @duportal
  • 2026-08-30:
    • [release.ci] Fastly API token expires => @duportal
    • [trusted.ci] Fastly API token expires => @duportal
  • 2026-08-31:
    • [infra.ci] Azure Credential to deploy docs.jenkins.io expires => @7JoquqrvQFu6FDlR14JVhg
  • 2026-09-03:
    • [infra.ci] Azure AD Application password for updatecli expires => @7JoquqrvQFu6FDlR14JVhg
    • [infra.ci] Azure Credential to write to reports.jenkins.io expires => @7JoquqrvQFu6FDlR14JVhg
    • Terraform credentials expire (aws-sponsored, azure, azure-net, cloudflare, datadog, digitalocean, fastly) => @dduportal
• Next major event: N.A.

Cloud Budgets

• Azure (CDF paid)

  • May: ~$2,800
  • June: ~$2,108
  • July: $2,182.84
  • August: $1226 (forecasted at $2.2k)
    • We will meet our threshold of $2.5k monthly
    • Note: publick8s should move to sponsored subscription ($993 for June, forecasted at $967 for August)

• Azure Sponsored subscription (Microsoft Credits): $75449 credits on August 1 2026

  • May: ~$5,734
  • June: ~$5,542
  • July: $5,518.05
  • August: $3,596 (forecasted at $6.3k but should be less as we only had a 1-day peak on cert.ci)
    • At current rate, we shall deplete credits on September 2027 (unless we have to move ci.jenkins.io here)

• DigitalOcean - Remaining $5,841.64 until January 02, 2027

  • May: $1186.97
  • June: $740.32
  • July: $496.88
  • August: $213 (forcasted at ~$390)
    • At current rate, we shall deplete credits on June 2027

• AWS:

  • Sponsored account, credits left: $29.8k until 01/31/2028 (on 1st July)
    • May: $14.9k
    • June: ~$7,893
    • July: $5,891.74
    • August: $3,368, -3% compared to last month(forecasted at $5.9k)
      • At current rate ($5.9k monthly), we shall deplete credits on January 2027
      • Noticeable increase this month

• JFrog Artifactory Usage

  • Storage: 1.94 Tb (increased)
  • Bandwidth:
    • May: 29.98 Tb
    • June: 23.24 Tb
    • July: 31.3 TB
    • August: 19.72 TB (forecasted at ~35Tb)

• Algolia API usage

  • Context: [plugins.jenkins.io] Watch Algolia search usage for June 2026
  • plugins.jenkins.io:
    • (July): ~41.1K (10%) search requests(max. 400K monthly)

Notes

• Done:

  • support
    • [agent-2.trusted.ci.jenkins.io] less than 20% disk space available on /
    • Request for PR preview to be enabled for contributor-spotlight repo’s non-default branch
    • Request for PR preview to be enabled for stories repo’s non-default branch
    • Enable previews for the plugin-modernizer-stats repo PRs
  • keep-infra-sane-and-maintainable
    • New 10G connected mirror sponsor - The Netherlands
  • keep-infra-up-to-date
    • [trusted.ci.jenkins.io] RPU Artifactory API token expires on the 2026-08-22

• Work in Progress:

  • keep-infra-sane-and-maintainable

    • Decrease BOM costs by splitting tests more efficiently
      • SplitTest in production since last week (used by last release!). 20 agents instead of hundreds, for the same (more or less) build duration => no more controller contention.
      • Following costs now (not sure if increase/decrease but necessary foundation for next steps)
      • Not directly infra, but enabled by splittests: optimization on reruning only failed tests or specified tests for PR tests, which will definitively decrease costs
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Wip on adding PostgreSQL on census.jenkins.io VM (infra + Puppet)
      • Goal: integrate May 2025 from census (so no more bus factor). Automation will come next.
    • [pipeline-library] new hadolint breaks lint on 6 docker repos
      • Recent hadolint release added new rules breaking linting while not needed for us.
      • Fixed on 1 repo, WiP on ignoring these new rules globally
    • [pipeline-library] define a common nodeWithTimeoutAndRetry() function for website builds
      • On hold until Damien is back
    • [cert.ci.jenkins.io] migrate controller VM from Azure CDF to Azure sponsored subscription
      • On hold until Damien is back

  • keep-infra-up-to-date

    • JDK patch upgrade campaign (July 2026)
      • Almost all done on infra except JDK8 (WiP on packer image, should be released later this week)
      • docker-packaging: changing the JDK install so we can controller it
      • Core: weekly ok, will wait for next LTS (Sept. 02) to close
    • Prepare update-center root CA rotation
      • Pinged Daniel, he will integrate the new certificate
    • Bump NodeJS from 22 to 24 (most recent LTS)
      • All done, except uplink. Back to triage
    • Drop Windows 2019 support
      • Support dropped from docker-agent (soon to be released)
      • Remaining docker-ssh-agent (jenkins image) and winp (CI pipeline only)
      • Blog post => @en3hD3iMRx6_6IXLNY0Rag
      • Then: packer-image (and Azure galleries) removal

• Issues added to the next milestone:

  • Support Mirror Selection in Update Center
    • Also contact India-located mirrors as they look out of date since 20 days

• Issues staying in backlog/triage:

  • [ci.jenkins.io] optimize Maven Cache loading during EC2 Agent startup
  • Migrate publick8s AKS cluster to the Azure 2026 Sponsored subscription
  • [Maven Dependency] could not be resolve org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 Maven plugin
  • Upgrade to Kubernetes 1.35
  • Debian upgrade campaign: bookworm (12.x) to trixie (13.x)
  • Tombstone plugin-site-api service (in favor of a static report file)
  • [Terraform azure/aws/shared-tools] Migrate Kubernetes provider resources (kubernetes_*) to versioned resources ( kubernetes_*_v1)
  • See if we can get OIDC enabled on our artifactory
  • main branch Jenkins build status is in progress but not build was triggered
  • msbuild not available on ci.jenkins.io Windows agent for winp build process
  • Sporadic HTTP 503 First byte timeout errors on jenkins.io
  • Provide Maven 4 pre-release templates on ci.jenkins.io
  • Move docker controller images publication job from trusted.ci.jenkins.io to release.ci.jenkins.io
  • Enable Jenkins CLI
  • [(status|contributors|stats|docs|stories|cn|jenkins-io-components).jenkins.io, Websites PR previews] Netlify billing plan changed: evaluate and act
  • Prepare for post-Jira account management
  • Inform of Ingress NGINX Retirement in March 2026
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • private docker image registry for staging core security releases
  • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
  • [PHS/Datadog] improve monitoring using application logs
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • Add monitoring for CD secrets updates
  • Matomo github/docker repos
  • Add reminder about Jira notification email address
  • [INFRA-3046] Monitor Jenkins mirrors Age
  • [INFRA-2352] Add monitoring for age of crawler files