Infrastructure Team Meeting - March 04, 2025

[dduportal (Damien Duportal)]

2025 03 04 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @hlemeur (Hervé Le Meur)

Announcements

1. Jenkins Weekly Releases

   • Last week: 2.499 succesfully released last week, after a first build failed due to formatting issues (fixed by core maintainers) - ref. You're invited to talk on Matrix
   • This Week: 2.500 is a security release, arriving Wednesday 5 Mar 2025

2. Announcements:

   • GitHub outage (API / Rate Limit) yesterday: GitHub Status - Incident with Issues, Git Operations and API Requests which caused ci.jenkins.io build issues

Upcoming Calendar

• Next Weekly:
  • 2.500, (tomorrow) 5 March 2025 - security release
  • 2.501, 11 March 2025
• Next LTS: (tomorrow) 2.492.2, 5 March, Release lead is Kris Stern - security release
  • Next baseline will be selected March 19
• Next Security Release as per jenkinsci-advisories: 2.500 and 2.492.2
• Upcoming credentials expirations (~3 weeks):
  • 7 March 2025:
    • infra.ci.jenkins.io’s Netlify token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] Netlify token expires the 09 December 2024 · Issue #4446 · jenkins-infra/helpdesk · GitHub)
    • infra.ci.jenkins.io’s NPM token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] NPM token expires the 7 December 2024 · Issue #4447 · jenkins-infra/helpdesk · GitHub)
• Next major event:
  • SoCal happens this week: Mark will be OOO because attending, with Basil and Alyssa

Cloud Budgets

• Azure CDF:

  • December: $4,4k (invoice)
  • January: $4.3k (invoice)
  • February: $4.1k
  • March: $462 (forecast at 4k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $47,180 until 31 May 2025

  • December: $9,5k
  • January: $13,1k
  • February: $11.2k
  • March: $487 (forecast at 3.8k)

• DigitalOcean - Remaining $14.5k until January 02, 2026

  • December: $192 (invoice)
  • January: $219 (invoice)
  • February: $237 (invoice)
  • March: $29 (forecast at $230)

• AWS:

  • CloudBees:
    • December: $540
    • January: $543
    • February: $550
    • March: $47.55 (forecast at $584)
  • Sponsored account (~$51k credits lefts until 01/31/2027)
    • December: $595
    • January: $1.4k
    • February: $8.5k
    • March: $2.2k (forecast at $13.5k)

• Jfrog Artifactory Usage (TBD)

Notes

• Done:

  • Support:

    • GitHub rate limit error
    • forgot my login

  • Keep platform up to date:

    • [packer-images] fail to build on main for the amazon windows 2019 fast-launch related
    • Update-center / crawler root certificate expires the 03th of April 2025 (failing job the 3th of March 2025)

• Closed as not planned:

  • Redirect Chinese pages to English pages and shutdown the Chinese site
  • Herbert Ugwu
  • Not able to login to artifactory

• Work in Progress:

  • Support:

    • docker-ssh-agent builds consistently timeout on ci.jenkins.io
      • Short term: let’s disable the failing test and release a new version (trusted not impacted)
      • Issue is correlated to EOF errors during layer extraction. Could be Mirror registry and/or Windows VM agent EBS disk. Need more analysis
    • Deploy jenkins-prototype on Netlify
      • First step: set up website in Netlify with automatic build (still in netlify) for the principal branch
      • Second step: ask Jan if PR preview are needed (Netlify can handle it) and implement it if yes
    • New repo scoop-bucket for app manifest distribution
      • Stephane need help from a GH jenkins-infra admin (missing permissions for GH APP)
      • Team check up to be done
    • https://updates.jenkins.io/stable/latest/jenkins.war returns 404
      • Should be an Apache redirection to the latest version. Make sense. Might be an unforseen (minor) consequence of the new UC

  • Keep platform up to date:

    • Enhancements possible for packer-image pipeline
      • Work on the pipeline, wip by Stephane.
    • [pipeline-library/updatecli] Allow custom version and execution in current “pipeline node” context
      • 2 sub tasks: specify a custom version, and then specify a “no node”
      • Unit tests are good, integration tests for the upcoming week

  • Reduce Artifactory storage and bandwidth use:

    • [repo.jenkins-ci.org] Ensure repository incrementals is automatically garbage-collected
      • Darin’s script is running regularly, let’s watch it and its effects
    • Reduce Artifactory storage and bandwidth use
      • Still have increasing storage usage: need to find the cause
      • Need to prepare jcenter-cache cleanup

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • Runbook updated
    • PRs for cleanup opened: to be merged and deployed for effective cleanup in Azure

  • [INFRA-2851] Add “Delete account” to accounts.jenkins.io

    • Back to backlog (subtask of accounts.jio replacement)

• Triage (New Issues and delayed issues):

  • Delayed:

    • [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents
      • Need cleanup of Karpenter part
    • [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM with Job Cacher
      • Damien need to write the plan and start the cache evaluation
    • Instability of artifact-caching-proxy on AWS
    • New mirror in India
      • 3 new mirrors to be added \o/
      • PR is ready (runbook, with the addon commands. Not as code.)
        • Also runbook misses the backup command which is needed => @dduportal
      • Planning for Friday (morning EU)
    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
    • CD release of pipeline-maven failed with 403
    • [CD / GitHub Actions] Reached Docker.io rate limit
    • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
      • Delayed (again)

  • Triage:

    • Plugin BOM release builds on AWS ci.jenkins.io fail much more frequently

      • Add to milestone
      • Need to check metrics (what is the blocker at resource level?)

    • Windows agents have different character set than in the past

      • Add to milestone
      • not important

    • Transfer casino-number-guessing-game Repository to jenkins-docs Organization

      • Add to milestone

    • Add a real-world job to weekly.ci.jenkins.io

      • Add to milestone

    • Upgrade to Kubernetes 1.31

    • Move collection of stats out from Kohsuke’s home

    • Support [skip ci] on default branch

    • Create build for jenkinsci/winp on release ci server

    • [Update Center] HTTP/404 on /current/updates/*.json* links

    • dnf5 update fails with gpgcheck=1

    • build failure with useArtifactCachingProxy=true and dependency with version range

    • External user struggling to submit story to stories.jenkins.io

    • Add monitoring for CD secrets updates

    • Switch agent (java home) to JDK21 default

    • Switch default JDK to 21 for pipeline libraries

    • Switch default JDK to 21 for build tools

    • Move controllers to JDK21 (runtime)

    • Move agents to JDK21 (runtime)

• ToDo (next milestone) (infra-team-sync-2025-03-11 Milestone · GitHub)