Infrastructure Team Meeting - September 01, 2026

[dduportal (Damien Duportal)]

2026 09 01 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @lemeurherve (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.579 went well
   • This Week: 2.580 will happen tomorrow (advisory)
2. Announcements:
   • Team capacity:
     • Hervé and Damien are back
     • Jay and Mark are off an come back tomorrow
   • Priorities:
     • Infra Roadmap: Jenkins Roadmap
     • Current EPICs (by priority): GitHub · Where software is built
     • Topic worth mentioning (timeframe of the upcoming 2 months)
       • mirrorbits v0.6.2 (from 0.6.1)
       • Nginx ingress retirement
       • Kubernetes 1.35 upgrade
     • Sponsoring: NodeOPS
       • Damien as a test account and a few credits granted for tests. Starting to test their plugin (still private source).

Upcoming Calendar

• Next infra meeting: 2026-09-08
• Next Weekly: 2026-09-02 (tomorrow - advisory) - 2.580
  • 2026-09-08 - 2.581
• Next LTS: 2026-09-02 (tomorrow - advisory) - 2.568.3
  • Backports: Prepare release, packaging and docker backports for LTS 2.568.3
• Next Security Release as per jenkinsci-advisories:
  • https://groups.google.com/g/jenkinsci-advisories/c/rlFfbtAoyQo/m/01Xfv1-uAgAJ
• Upcoming credentials expirations (~3 weeks):
  • 2026-09-03:
    • Terraform credentials expire (aws-sponsored, azure, azure-net, cloudflare, datadog, digitalocean, fastly) - [Terraform Credentials] Expirations of multiple credentials (backends, API) in jenkins-infra/terraform-states on 2026-09-03 · Issue #5273 · jenkins-infra/helpdesk · GitHub
• Next major event: N.A.

Cloud Budgets

Not reporting for September as we are the 1st (half a day == senseless data)

• Azure (CDF paid)

  • May: ~$2,800
  • June: ~$2,108
  • July: $2,182.84
  • August: $2,172.17
    • Max. threshold of $2.5k monthly met
    • Note: publick8s should move to sponsored subscription (~$990 monthly)

• Azure Sponsored subscription (Microsoft Credits): $75449 credits on August 1 2026

  • May: ~$5,734
  • June: ~$5,542
  • July: $5,518.05
  • August: $6,652.23
    • Increase due to cert.ci.jenkins.io builds for tomorrow’s advisory
    • At current rate, we shall deplete credits on August 2027 (unless we have to move ci.jenkins.io here)

• DigitalOcean - Remaining $5,671.59 until January 02, 2027

  • May: $1186.97
  • June: $740.32
  • July: $496.88
  • August: $382.55
    • At current rate, we shall deplete credits on June 2027

• AWS:

  • Sponsored account, credits left: $29.8k until 01/31/2028 (on 1st July)
    • May: $14.9k
    • June: ~$7,893
    • July: $5,891.74
    • August: $4,364.68 -19% compared to last month
      • At current rate ($5.6k monthly), we shall deplete credits on January 2027
      • Renewal: new contacts at AWS (Mila Z. is changing role)

• JFrog Artifactory Usage

  • Storage: 1.8 Tb (decreased - GC of incremental-publisher)
  • Bandwidth:
    • May: 29.98 Tb
    • June: 23.24 Tb
    • July: 31.3 TB
    • August: 36,53Tb

• Algolia API usage

  • Context: [plugins.jenkins.io] Watch Algolia search usage for June 2026
  • plugins.jenkins.io:
    • (July): ~87.7K (21%) search requests(max. 400K monthly)

Notes

• Done:

  • keep-infra-up-to-date

    • Prepare release, packaging and docker backports for LTS 2.568.3
    • [trusted.ci.jenkins.io] Fastly token used to decache www.jenkins.io expires on 30 August 2026
    • [release.ci.jenkins.io] Fastly token used to decache pkg.jenkins.io expires on 30 August 2026
    • [trusted.ci.jenkins.io] Update Center credential for Cloudflare R2 expires on 2026-08-27
    • [infra.ci.jenkins.io] Fastly API tokens to deploy [infra.ci] Fastly Automation token used to purge (plugins|contributors|spotlight).jenkins.io websites expires on 26 August 2026
    • [infra.ci.jenkins.io] Fastly API tokens used by the Fastly Terraform project expires on 26 August 2026

  • support

    • [Plugin Health Scoring] Getting latest logs of the application
      • Not updated since a long time! Fix pushed and deployed but PHS frontend seems to send 503.
      • We have to check the monitoring as well => @hervelemeur to check if we have monitoring in place
    • Jira is responding very, very slowly

  • keep-infra-sane-and-maintainable

    • [cert.ci.jenkins.io] Set up a “common” Resource Group and subnet with shared resources in the sponsored subscription

• Work in Progress:

  • keep-infra-sane-and-maintainable

    • New Jenkins mirror in Romania — ZEROLAG
      • Re-add the mirror then do a rollback to check if still down => @hervelemeur
      • Check runbook updates, need backup YAML too => @hervelemeur
    • [pipeline-library] define a common nodeWithTimeoutAndRetry() function for website builds
      • PoC (LLM generated) working locally, need to check on infra.ci + unit tests
    • [cert.ci.jenkins.io] migrate controller VM from Azure CDF to Azure sponsored subscription
      • Resuming work, need a “task list” updated => @dduportal to update, then team discussion on who takes care of it
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • Damien: set up census to have pgsql, golang, SSH access to usage, retrieve and loaded dump from andrew
      • Herve: required to install make, built the CLI successfully, retrieved the May/June 2025 logs, successfully loaded into pgSQL and we have May 2025 stats CSV generated from pg already pushed!
      • Some timings:
        • 1 month copy (with rsync) from usage is ~3 GB (5-10 min)
        • 1 month of logs imported in pg takes ~8h
        • Generation of a 1 month report: takes ~4,5 hours
      • !! Last usage stats is March 2026 => we need to contact KK
      • Disk usage to be checked
      • GC/archive?
      • Need to report on issue/runbook (whatever)

  • keep-infra-up-to-date

    • JDK critical patch upgrade campaign (August 2026)
      • Oracle and Adoptium will be monthly updated from now.
    • Prepare update-center root CA rotation
      • On hold: waiting for Daniel’s PR to be merged.
      • Then we’ll evaluate the update center integration and timings
    • Drop Windows 2019 support
      • Waiting for docker-ssh-agent release
      • Then winp (ci.jenkins.io) and packer-images cleanup

  • support

    • Create sonarcloud-token
      • @dduportal need to answer and the team will check the sonarcloud access to generate the token
    • HealthScore under Security shows 0% for a resolved security issue
      • Adrien has the logs and work on fixes. Herve follows up with him
    • Support Mirror Selection in Update Center
      • We have not heard from Tuna mirror.
      • But it is the only China-based mirror
      • Should we run a mirror in China by ourselves? Let’s bring this to the board.

• Issues staying in backlog/triage:

  • Bump NodeJS from 22 to 24 (most recent LTS)
  • [ci.jenkins.io] optimize Maven Cache loading during EC2 Agent startup
  • Migrate publick8s AKS cluster to the Azure 2026 Sponsored subscription
  • [Maven Dependency] could not be resolve org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 Maven plugin
  • Upgrade to Kubernetes 1.35
  • Debian upgrade campaign: bookworm (12.x) to trixie (13.x)
  • Tombstone plugin-site-api service (in favor of a static report file)
  • [Terraform azure/aws/shared-tools] Migrate Kubernetes provider resources (kubernetes_*) to versioned resources ( kubernetes_*_v1)
  • See if we can get OIDC enabled on our artifactory
  • main branch Jenkins build status is in progress but not build was triggered
  • msbuild not available on ci.jenkins.io Windows agent for winp build process
  • Sporadic HTTP 503 First byte timeout errors on jenkins.io
  • Provide Maven 4 pre-release templates on ci.jenkins.io
  • Move docker controller images publication job from trusted.ci.jenkins.io to release.ci.jenkins.io
  • Enable Jenkins CLI
  • [(status|contributors|stats|docs|stories|cn|jenkins-io-components).jenkins.io, Websites PR previews] Netlify billing plan changed: evaluate and act
  • Prepare for post-Jira account management
  • Inform of Ingress NGINX Retirement in March 2026
  • [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  • [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  • private docker image registry for staging core security releases
  • [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
  • [PHS/Datadog] improve monitoring using application logs
  • Chinese jenkins site incorrect site redirection
  • Add a real-world job to weekly.ci.jenkins.io
  • [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
  • Create build for jenkinsci/winp on release ci server
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • Add monitoring for CD secrets updates
  • Matomo github/docker repos
  • Add reminder about Jira notification email address
  • [INFRA-3046] Monitor Jenkins mirrors Age
  • [INFRA-2352] Add monitoring for age of crawler files

• Issues added to the next milestone:

  • JDK patch upgrade campaign (July 2026)
  • Decrease BOM costs by splitting tests more efficiently
    • Herve will check costs to see if see formal proof or not. But we decreased
    • Next step: implement “skip” tests already passing on previous builds on PRs (or principal branch if same commit, but to be confirmed)