Infrastructure Team Meeting - March 25, 2025

[dduportal (Damien Duportal)]

2025 03 25 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @smerle33 (Stéphane Merle)
• Giovanni Vaccarino (GSoC future mentee)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.502 released succesfully - You're invited to talk on Matrix
   • This Week: 2.503 in progress - You're invited to talk on Matrix
2. Ingress Nginx CVE (CVE-2025-1974 - Ingress-nginx CVE-2025-1974: What You Need to Know | Kubernetes)
   • We were relatively safe as our clusters with an ingress are not multitenant
   • Patched 6 hours ago
   • Additionally: we disabled webhooks admission
   • Important: Ingress Nginx will enter maintenance mode end of 2025: ⚠️ Ingress NGINX Project Status Update ⚠️ · Issue #13002 · kubernetes/ingress-nginx · GitHub. We’ll have to look at the Gateway API in Kubernetes.
3. We had a plugin security advisory last week: https://groups.google.com/g/jenkinsci-advisories/c/91u0Sc0WTOY

Upcoming Calendar

• Next Weekly: 2025-04-01 - 2.504
• Next LTS: 2.492.3, Wednesday April 2, Bruno is the release lead, mentored by Kris Stern
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2025-04-08:
    • Azure SP for www.jenkins.io on trusted.ci.jenkins.io (to deploy website) - Azure File Share Principal `www.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #964 · jenkins-infra/azure · GitHub
    • Azure SP for javadoc.jenkins.io on trusted.ci.jenkins.io (to deploy website) - Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #963 · jenkins-infra/azure · GitHub
    • Azure SP for GeoIP staging storage on publick8s (to mount Azure File volume) - Azure File Share Principal `geoip_staging` on `publick8s` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #962 · jenkins-infra/azure · GitHub
    • Azure SP for GeoIP (production) storage on publick8s (to mount Azure File volume) - Azure File Share Principal `geoip` on `publick8s` expires on `2025-04-08T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #961 · jenkins-infra/azure · GitHub
    • Azure SP for docs.jenkins.io on infra.ci.jenkins.io (to deploy website) - New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: 2025-04-08T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #960 · jenkins-infra/azure · GitHub
• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • December: $4,4k (invoice)
  • January: $4.3k (invoice)
  • February: $3,9k (invoice)
  • March: $3,372 (forecast at 4.3k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $44,248 until 31 May 2025

  • December: $9,5k
  • January: $13,1k
  • February: $11.2k
  • March: $3,425 (forecast at 4.3k)

• DigitalOcean - Remaining $14,6k until January 02, 2026

  • December: $192 (invoice)
  • January: $219 (invoice)
  • February: $237 (invoice)
  • March: $214 (forecast at $265)

• AWS:

  • CloudBees:
    • December: $540
    • January: $543
    • February: $550
    • March: $424 (forecast at $545)
  • Sponsored account (~XXXk credits lefts until 01/31/2027)
    • December: $595
    • January: $1.4k
    • February: $8.5k
    • March: $13,4k (forecast at $15.2k)
      • Ref. [aws.ci.jenkins.io] High EC2 costs on the USE2-NatGateway-Bytes and USE2-DataTransfer-Regional-Bytes
      • Data transfer through NAT Gateways / Inter AZ data transfer have been contained. We could work on the ECR eventually
      • Instance costs (spot everywhere!)

• Jfrog Artifactory Usage

  • Storage: 3.69 TB (steady)
    • Darin’s cleanup helps to keep us below the 5 TB threshold
    • Next step would be jcenter removal. April?
  • Bandwidth still high though (> 10 Tb threshold)
    • March: 36.77 Tb
      • A lot of variations: some days at 1 Tb, others at 280Gb

Notes

• Done:

  • Support:

    • Remove olivergondza a Jenkins JIRA component lead
    • Terraform Datadog project fails to run on the principal branch
    • Request to be added into jenkinsci organization owners
    • [packer-images] logs within windows agents on amazon
    • New mirror in India
    • CD release of pipeline-maven failed with 403
    • Instability of artifact-caching-proxy on AWS
    • The UpdateCli step fails regularly when processing jenkins.io PRs

  • Keep platform up to date:

    • Terraform DigitalOcean build fails (Terraform 1.11.x related)
    • [cert.ci.jenkins.io, trusted.ci.jenkins.io] Azure Principal used for Let’s Encrypt DNS challenges expires on 2025-03-23
    • [Terraform Backends] Expiration of credentials for Backend States and Cloudflare API tokens the 2025-03-23
    • [pipeline-library/updatecli] Allow custom version and execution in current “pipeline node” context

  • AWS costs:

    • [ci.jenkins.io] set up S3 Artifact Manager for artifacts/stash/unstash (instead of controller file system)

• Work in Progress:

  • Support:

    • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
      • Step 1: Draft PR
      • Step 2: in order to test, you’ll need a new Multibranch pipeline job in infra.ci
    • [ci.jenkins.io] Docker pull fails with random ERROR: failed to read expected number of bytes: unexpected EOF
      • Need to set TTL of cached layers to zero
      • And also we have to cleanup the cached layers to remove corrupted ones
    • Several Jenkins core tests on ci.jenkins.io fail more often since transition to AWS
      • Need to determine in which kind of agents these tests are run (and failing)
    • docker-ssh-agent builds consistently timeout on ci.jenkins.io
      • On hold (AWS costs + local storage)
    • Deploy jenkins-prototype on Netlify
      • Work very well! One last PR introducing Netlify as code (instead of manual settings)

  • Keep platform up to date:

    • Upgrade to Kubernetes 1.31
      • kubectl 1.31 is used (client side)
      • Issue updated to anticipate planning
    • [Upgrade Campaign] Bump Cloudflare Terraform provider to 5.x
      • Pinned to 4.x as the current 5.1.x is broken and almost destroyed the Update Center infra

  • [aws.ci.jenkins.io] High EC2 costs on the USE2-NatGateway-Bytes and USE2-DataTransfer-Regional-Bytes

    • [ci.jenkins.io] Use Spot instances for VM agents to decrease costs
      • Moving as much as “On Demand” (BoxUsage) EC2 instances to Spot
      • Implemented “retry” on buildPlugin pipeline library
      • ci.jenkins.io: all Linux EKS nodes (container agents), and all EC2 VM agents are now using spot instances
        • We selected spot with a <5% (the lowest range) interruption frequency for EC2 VM agents (range was 15-20% with former instance type)
        • Karpenter is expected to select the lowest interruption range: let’s see if it works as expected
      • We still have a few “non spot” instances:
        • We have ci.jenkins.io’s VM, and the 2 “EKS application nodes”
        • packer: to be checked (and done if not spot) => @smerle
        • Fast Launch? (=> @dduportal)
      • We have to inform developers we switched fully to Spot, so they can report to us in case of a problem => @smerle
    • [ci.jenkins.io] Use EC2 local store (local NVMe instead of EBS network storage)
      • Instance types for EC2 VM agents have been changed to be compliant with instance store
      • WiP on Linux instances with cloudinit to format and mount it into the agent workspace
    • [ci.jenkins.io] Fix HTTP/429 DockerHub errors for buildx builder using docker-container
      • Should help, taking over from Herve as no news
    • [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM
      • Caching works very well
      • We now have a weekly job which generate cache from scratch
      • Last step: monitor the cache archive (less than 7 days and more than 1Gb)

  • Reduce Artifactory storage and bandwidth use

    • [repo.jenkins-ci.org] Ensure repository incrementals is automatically garbage-collected
    • Resuming work in April once Darin will be back

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud$

    • Started cleaning up Puppet
    • Preparing jenkins.io blogpost announcing UC force to TLS

  • On hold (low priority):

    • Build failed to resume
    • Enhancements possible for packer-image pipeline
    • Add a real-world job to weekly.ci.jenkins.io
      • Resumed after Kube 1.31
    • https://updates.jenkins.io/stable/latest/jenkins.war returns 404
      * Need to cleanup UC leftovers
    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
      • One last week waitinf from Andrew
    • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
      • Delayed

• New Issues (to triage):

  • Delayed issues:
    • Move collection of stats out from Kohsuke’s home
    • Support [skip ci] on default branch
    • Create build for jenkinsci/winp on release ci server
    • [Update Center] HTTP/404 on /current/updates/*.json* links
    • dnf5 update fails with gpgcheck=1
    • build failure with useArtifactCachingProxy=true and dependency with version range
    • External user struggling to submit story to stories.jenkins.io
    • Add monitoring for CD secrets updates
    • Switch agent (java home) to JDK21 default
    • Switch default JDK to 21 for pipeline libraries
    • Switch default JDK to 21 for build tools
    • Move controllers to JDK21 (runtime)
    • Move agents to JDK21 (runtime)

• ToDo (next milestone) (infra-team-sync-2025-04-01 Milestone · GitHub)

  • ERR_CERT_COMMON_NAME_INVALID on https://ci.jenkins-ci.org
    • Low priority
    • But might be usefull to allow multiple SAN on the ci.jenkins.io Puppet Let’s Encrypt system
  • S3 data dump of selected ci.jenkins.io data for GSoC 2025 LLM project
    • New milestone
    • Scope defined: we only have to export Job data (pipeline logs)