Infrastructure Team Meeting - September 08, 2026

[dduportal (Damien Duportal)]

2026 09 08 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @lemeurherve (Hervé Le Meur)
• Robson

Announcements

1. Jenkins Weekly Releases

   • Last Week: 2.580 went well
     • Note: there was a mistake for the Docker tag which wasn’t created from a commit of the master branch. Will be eclipsed by 2.581 but we might want to sync with jenSec team to be sure we don’t do the same next time. LTS 2.580 LTS line in jenkinsci/docker has to be created from a different commit.
   • This Week: 2.581, went well

2. Announcements:

   • Team capacity:
     • Jay is off today (sick days), not sure when he’ll be back
   • Priorities:
     • Infra Roadmap: Jenkins Roadmap
       • Deserve an update (stats and upcoming EPICs)
     • Current EPICs (by priority): GitHub · Where software is built
       • Closed AWS-billing and opened [Autumn/2026] Determine and set up the future Cloud location of `ci.jenkins.io` before EoY 2026 · Issue #5283 · jenkins-infra/helpdesk · GitHub for ci.jenkins.io
     • Topic worth mentioning (timeframe of the upcoming 2 months)
       • mirrorbits v0.6.2 (from 0.6.1) - [get.jenkins.io/updates.jenkins.io] Bump mirrorbits from `0.6.1` to `0.6.2` · Issue #5279 · jenkins-infra/helpdesk · GitHub
       • Nginx ingress retirement - Inform of Ingress NGINX Retirement in March 2026 · Issue #4869 · jenkins-infra/helpdesk · GitHub
       • Kubernetes 1.35 upgrade - Upgrade to Kubernetes 1.35 · Issue #5168 · jenkins-infra/helpdesk · GitHub
     • Sponsoring: NodeOPS
       • Damien as a test account and a few credits granted for tests. WiP on testing their plugin.

Upcoming Calendar

• Next infra meeting: 2026-09-15
• Next Weekly: 2026-09-15 - 2.582
• Next LTS: 2026-09-30 - 2.580.1
  • RC: 2026-09-12
    • A bit earlier than usual as Mark (release lead) won’t be present the week after
  • Backports preparation helpdesk issue: to be opened
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks): N.A.
• Next major event: N.A.

Cloud Budgets

• Azure (CDF paid)

  • May: ~$2,800
  • June: ~$2,108
  • July: $2,182.84
  • August: $2,179,52
  • September: $505 (forecast at ~2.0)
    • Max. threshold of $2.5k monthly met
    • Note: publick8s should move to sponsored subscription (~$990 monthly)

• Azure Sponsored subscription (Microsoft Credits): $68,697 credits on September 1 2026 (expire on Oct. 2027)

  • May: ~$5,734
  • June: ~$5,542
  • July: $5,518.05
  • August: $6,652.23
    • Increase due to cert.ci.jenkins.io builds for tomorrow’s advisory
  • September: $1,463 (forecast at $6280)
    • At current rate, we shall have 11 month (at 6.2k/monthly), depleting credits on August 2027 (unless we have to move ci.jenkins.io here)

• DigitalOcean - Remaining $5,547.37 until January 02, 2027

  • May: $1186.97
  • June: $740.32
  • July: $496.88
  • August: $382.55
  • September: $124.22 (forecast at $465)
    • At current rate, we have ~12 months left

• AWS:

  • Sponsored account, credits left: $24,424.79 until 01/31/2028 (on 1st September)
    • May: $14.9k
    • June: ~$7,893
    • July: $5,891.74
    • August: $5,398.51 -19% compared to last month
    • September: $1,234.85 (-28% than same period last month) (forecasted at $4,7778 | -11% decrease )
      • At current rate ($5.4k monthly), we shall deplete credits on January 2027
      • No more cost reduction effort: from now we focus on either getting more credits or moving to Azure (or somewhere else) - [Autumn/2026] Determine and set up the future Cloud location of `ci.jenkins.io` before EoY 2026 · Issue #5283 · jenkins-infra/helpdesk · GitHub

• JFrog Artifactory Usage

  • Storage: 1.8 Tb (decreased - GC of incremental-publisher)
  • Bandwidth:
    • May: 29.98 Tb
    • June: 23.24 TB
    • July: 31.3 TB
    • August: 36.53 TB
    • September: 14.71 TB (forecast: 55 TB)

• Algolia API usage (max. 400K monthly)

  • Context: [plugins.jenkins.io] Watch Algolia search usage for June 2026
  • plugins.jenkins.io:
    • July: 87.7k search requests (21%)
    • August: 89.1k search requests (22%)
    • September: 16.5k search requests (4%)

Notes

• Done:

  • keep-infra-up-to-date

    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.568.3
    • [Terraform Credentials] Expirations of multiple credentials (backends, API) in jenkins-infra/terraform-states on 2026-09-03
    • JDK critical patch upgrade campaign (August 2026)
    • JDK patch upgrade campaign (July 2026)

  • support

    • Newly hosted plugin not discovered by ci.jenkins.io

  • keep-infra-sane-and-maintainable

    • New Jenkins mirror in Romania — ZEROLAG

      • Important: we have to update our GeoIP database because it’s seen as located in China. Short term: set it to country only to bypass

    • [cert.ci.jenkins.io] Set up a “common” Resource Group and subnet with shared resources in the sponsored subscription

    • Decrease BOM costs by splitting tests more efficiently

      • Difficult to measure build costs (no thanks AWS). Overall cost decreased: more compute costs byt way less “other costs”.

      • From June 2026 to July 2026 total costs in AWS decreased by 25.3%, and from July 2026 to August 2026 they decreased by another 8.5%.

        That is a total decrease of 31.6% since the introduction of split tests

    • [cert.ci.jenkins.io] migrate controller VM from Azure CDF to Azure sponsored subscription

      • Not a big gain for billing BUT we verified that we can have things running in Sweden

• Closed as not planned:

  • keep-infra-sane-and-maintainable
    • [pipeline-library] define a common nodeWithTimeoutAndRetry() function for website builds
  • support
    • Chinese jenkins site incorrect site redirection

• Work in Progress:

  • support

    • Cannot log in to Artifactory (repo.jenkins-ci.org) — incorrect username/password or locked user
      • Could it be new outbound IP not allowed in our LDAP? Check our automation
    • [ci.jenkins.io] DockerHub transparent mirror/proxy seems to be down, resulting in HTTP/429 from DockerHub
      • Let’s try to support custom registry to allow usage of AWS internal ECR Dockerhub mirror
    • Create sonarcloud-token
      • No SOT available. Waiting for Jensec and jenkinsci admin to validate the “technical GH user” proposal using a PAT
    • HealthScore under Security shows 0% for a resolved security issue
      • Waiting for Adrien
    • Support Mirror Selection in Update Center
      • Waiting for the user to get us a sponsor or contact
      • TUNA is disabled. Let’s wait until end of September before makign decisions as they don’t answer to our emails
      • We should work on building a mirror in China ourselves

  • keep-infra-up-to-date

    • Prepare update-center root CA rotation
      • Waiting for the new CA to be released in Jenkins Core. Stale.
    • Drop Windows 2019 support
      • Only remaining of Windows 2019 is the ci.jenkins.io job of jenkinsci/winp. @dduportal to move it to 2025 (vs code tooling)
      • Packer images don’t build 2019 anymore (no more updated)
      • cert.ci to be cleaned up (template agents)
      • Cleanup:
        • Azure galleries can be removed when cert.ci is cleaned up.
        • AWS to be done manually when winp is not using 2019 anymore

  • keep-infra-sane-and-maintainable

    • Optimise costs and maintainance by merging Windows 2022 and Windows 2025 templates
      • We already have docker and docker-agents project doing this (both AWS and Azure)
      • Next (wip) is docker SSH agent
      • Next steps:
        • Find if any other consumers (either projects or controller agent templates). At least release.ci.
        • Then: we can get rid of Windows 2022.
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
      • May and June 2025, visible on stats.jenkins.io
      • July in progress (@dduportal)
      • August by @7JoquqrvQFu6FDlR14JVhg then
      • Next steps:
        • automation up to march 2026.
        • We have to contact KK to resume March → up to today from his basement machine

• Issues added to the next milestone:

  • [ci.jenkins.io] Should we persist “Disable performance enhancements” git config?
  • Add buildWebsite pipeline library function
  • [get.jenkins.io/updates.jenkins.io] Bump mirrorbits from 0.6.1 to 0.6.2
  • Clean up autolink references
    • @timja is OK to run the cleanup script (as admin of jenkinsci GH org)
    • @hervelemeur to prepare the script
  • Migrate publick8s AKS cluster to the Azure 2026 Sponsored subscription
    • Migration to Sweden + sponsored subscription
    • @dduportal to make the plan
  • [Terraform azure/aws/shared-tools] Migrate Kubernetes provider resources (kubernetes_*) to versioned resources ( kubernetes_*_v1)
  • [get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update