Infrastructure Team Meeting - September 15, 2025

[lemeurherve (Hervé Le Meur)]

2026 09 15 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @lemeurherve (Hervé Le Meur)
• @jayfranco999 (Jay Reddy)
• Robson

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.581, went well
   • This Week: 2.582, completed
     • The “recalculate metadata” step of the release build failed, Damien completed the remaining steps manually
     • Issue opened: Artifactory responds HTTP/403 to requests for metadata recalculation
2. Announcements:
   • Team capacity: Damien will be off this Friday, Hervé next Monday and Tuesday
   • Priorities:
     • Infra Roadmap: Jenkins Roadmap
       • Deserve an update (stats and upcoming EPICs)
     • Current EPICs (by priority): GitHub · Where software is built
     • Topic worth mentioning (timeframe of the upcoming 2 months)
       • Nginx ingress retirement - Inform of Ingress NGINX Retirement in March 2026 · Issue #4869 · jenkins-infra/helpdesk · GitHub
       • Kubernetes 1.35 upgrade - Upgrade to Kubernetes 1.35 · Issue #5168 · jenkins-infra/helpdesk · GitHub
     • Sponsoring: NodeOPS
       • Damien has a test account and a few credits granted for tests. WiP on testing their plugin.

Upcoming Calendar

• Next infra meeting: 2026-09-22

• Next Weekly: 2026-09-22 - 2.583

• Next LTS: 2026-09-30 - 2.580.1

  • Backports helpdesk: Backport docker, packaging, and release changes for LTS 2.580.1 · Issue #5284 · jenkins-infra/helpdesk · GitHub

• Next Security Release as per jenkinsci-advisories: Wednesday, September 16, plugins only

  The Jenkins project will publish a security advisory for Jenkins plugins on Wednesday, September 16. The highest severity is ‘High’. The most popular included plugin is installed on more than 75% of known instances.

• Upcoming credentials expirations (~3 weeks):

  • 2026-10-07: Renew ci.jenkins.io API token for incremental-publisher

• Next major event: N/A

Cloud Budgets

• Azure (CDF paid)

  • May: ~$2,800
  • June: ~$2,108
  • July: $2,182.84
  • August: $2,179,52
  • September: $1003 (forecast at $2,067)
    • Max. threshold of $2.5k monthly met
    • Note: publick8s should move to sponsored subscription (~$990 monthly)

• Azure Sponsored subscription (Microsoft Credits): $60,446 credits on September 15 2026 (expire on Oct. 2027)

  • May: ~$5,734
  • June: ~$5,542
  • July: $5,518.05
  • August: $6,652.23
    • Increase due to cert.ci.jenkins.io builds for tomorrow’s advisory
  • September: $2,345 (forecast at $5287, previously at $6280)
    • At current rate, we shall have a bit more than 11 months (at 5.3k/monthly), depleting credits on August 2027 (unless we have to move ci.jenkins.io here)

• DigitalOcean - Remaining $5,435.31 until January 02, 2027 (on September 15)

  • May: $1186.97
  • June: $740.32
  • July: $496.88
  • August: $382.55
  • September: $236.28 (forecast at $472)
    • At current rate, we have ~12 months left

• AWS:

  • Sponsored account, credits left: $24,424.79 until 01/31/2028 (on 15th September)
    • May: $14.9k
    • June: ~$7,893
    • July: $5,891.74
    • August: $5,398.51 -19% compared to last month
    • September: $2,202.65 (-28% than same period last month) (forecasted at $4,221, previously $4,7778 | -22% decrease )
      • At current rate ($4.3k monthly), we shall deplete credits on mid-February 2027
      • No more cost reduction effort: from now we focus on either getting more credits or moving to Azure (or somewhere else) - [Autumn/2026] Determine and set up the future Cloud location of `ci.jenkins.io` before EoY 2026 · Issue #5283 · jenkins-infra/helpdesk · GitHub

• JFrog Artifactory Usage

  • Storage: 1.84 TB (was 1.8Tb last week)
  • Bandwidth:
    • May: 29.98 TB
    • June: 23.24 TB
    • July: 31.3 TB
    • August: 36.53 TB
    • September: 20.44 TB (forecast: 41 TB, previously 55 TB)

Notes

• Done:

  • support

    • Downgrade lemeurherve permission level on jenkinsci/jenkins from Write to Triage
    • Cannot log in to Artifactory (repo.jenkins-ci.org) — incorrect username/password or locked user
    • Clean up autolink references
    • HealthScore under Security shows 0% for a resolved security issue
    • msbuild not available on ci.jenkins.io Windows agent for winp build process

  • keep-infra-sane-and-maintainable

    • [ci.jenkins.io] Persist “Disable performance enhancements” git config as code

  • keep-infra-up-to-date

    • [get.jenkins.io/updates.jenkins.io] Bump mirrorbits from 0.6.1 to 0.6.2
    • [Terraform azure/aws/shared-tools] Migrate Kubernetes provider resources (kubernetes_*) to versioned resources ( kubernetes_*_v1)
    • Drop Windows 2019 support

• Closed as not planned:

  • support
    • publish maven dependencies for lts jenkins from repo.jenkins-ci.org to maven central
    • Publishing new plugin: AI Chatbot to Guide User Workflow

• Work in Progress:

  • support

    • [trusted.ci.jenkins.io] Crawler build is failing
    • 403 releasing plugin
    • [ci.jenkins.io] DockerHub transparent mirror/proxy seems to be down, resulting in HTTP/429 from DockerHub
      • Using public.ecr.com as prefix in front of all FROM directives (report links from closed PR to helpdesk issue)
    • Create sonarcloud-token
    • [get.jenkins.io] Ensure TUNA (China-based) mirror is reliable for Chinese users (original title: Support Mirror Selection in Update Center)
      • Brown testing until end of September
      • No response from them, no complain from users for now
      • We’ll remove it if still the same beginning November

  • keep-infra-sane-and-maintainable

    • Use a common buildWebsite pipeline library function for websites
    • Optimise costs and maintainance by merging Windows 2022 and Windows 2025 templates
    • [stats.jenkins.io/infra-statistics] Move “data for the usage stats site” generation (from anonymized data) out from Andrew machine
    • [Plugin Health Score] Monitor outcomes of the scores computation routines and alert us when failing for too long

  • keep-infra-up-to-date

    • Migrate publick8s AKS cluster to the Azure 2026 Sponsored subscription
    • Prepare update-center root CA rotation
    • [get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update

• Issues added to the next milestone:

  • Backport docker, packaging, and release changes for LTS 2.580.1

• Issues staying in backlog/triage:
  * [updates.jenkins.io] Get rid of Cloudflare R2 buckets for UC mirrors in favor of plain old VMs
  * Switch ci.jenkins.io git implementation from JGit to command line git
  * [ci.jenkins.io] optimize Maven Cache loading during EC2 Agent startup
  * [Maven Dependency] could not be resolve org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 Maven plugin
  * [packer-images] Bump Ubuntu base image from 22.04 to 26.04
  * Upgrade to Kubernetes 1.35
  * Debian upgrade campaign: bookworm (12.x) to trixie (13.x)
  * Tombstone plugin-site-api service (in favor of a static report file)
  * See if we can get OIDC enabled on our artifactory
  * main branch Jenkins build status is in progress but not build was triggered
  * Sporadic HTTP 503 First byte timeout errors on jenkins.io
  * Provide Maven 4 pre-release templates on ci.jenkins.io
  * Move docker controller images publication job from trusted.ci.jenkins.io to release.ci.jenkins.io
  * Enable Jenkins CLI
  * [(status|contributors|stats|docs|stories|cn|jenkins-io-components).jenkins.io, Websites PR previews] Netlify billing plan changed: evaluate and act
  * Prepare for post-Jira account management
  * Inform of Ingress NGINX Retirement in March 2026
  * [Azure Deprecation] Convert your OS disks to Standard SSD or Premium SSD before 8 September 2028
  * [Azure] Merge webservices data storage accounts into a single one with NFS v4.1
  * private docker image registry for staging core security releases
  * [ci.jenkins.io] diagnose and eventually fix the EC2 Windows VM performances and build failures/retries
  * [PHS/Datadog] improve monitoring using application logs
  * Add a real-world job to weekly.ci.jenkins.io
  * [ci.jenkins.io] Monitor and Garbage collect data volume of the DockerHub registry mirror and EC pull through cache
  * Move collection of stats out from Kohsuke’s home
  * Support [skip ci] on default branch
  * Create build for jenkinsci/winp on release ci server
  * [Update Center] HTTP/404 on /current/updates/*.json* links
  * Add monitoring for CD secrets updates
  * Support CD workflow on Windows
  * Matomo github/docker repos
  * Add reminder about Jira notification email address
  * [INFRA-3167] Move security settings to configuration-as-code for puppet managed instances
  * [INFRA-3046] Monitor Jenkins mirrors Age
  * [INFRA-2352] Add monitoring for age of crawler files

  • keep-infra-up-to-date
    • Bump NodeJS from 22 to 24 (most recent LTS)