Infrastructure Team Meeting - February 25, 2025

[smerle33 (Stéphane MERLE)]

2025 02 25 Jenkins Infra Meeting

Attendees

• @smerle33 (Stéphane Merle)
• @MarkEWaite (Mark Waite)
• @poddingue (Bruno Verachten)
• @jayfranco999 (Jay Reddy)

Announcements

1. Jenkins Weekly Releases

   • Last week: 2.498 released with no issues - ref. You're invited to talk on Matrix
   • This Week: 2.499 started on time but first build failed, new build relaunched, should be released and deployed later today - ref. You're invited to talk on Matrix

2. Announcements:

   • to follow up with ci.jenkins.io is now in AWS. Few performances issues and hiccups but overall work great (and Azure sponsor credits are not in danger anymore): we rollback to windows VMs instead of kubernetes pods as is was bugging some plugins (1 test of sse-gateway-plugin fails on ci.jenkins.io Windows agents · Issue #4540 · jenkins-infra/helpdesk · GitHub 5 tests of workflow-durable-task-plugin fail on ci.jenkins.io Windows agents · Issue #4541 · jenkins-infra/helpdesk · GitHub)
     • leading to the discussion to keep windows container or to embrace windows vm agent [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents · Issue #4554 · jenkins-infra/helpdesk · GitHub
       • to have the windows containers working we would have to solve problems [ci.jenkins.io] Paging file too small reported for config-file-provider and email-ext plugin builds · Issue #4552 · jenkins-infra/helpdesk · GitHub Paging file too small

Upcoming Calendar

• Next Weekly: 2.500, 4 March 2025
• Next LTS: 2.492.2, 5 March, Release lead is Kris Stern
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 3 March 2025: Update Center Signing Certificate expires
    • GitHub · Where software is built
    • Damien might be a SPOF: need to run it in pair with Stephane to ensure it is not the case (or it is acceptable)
      • he is as we need the CA certificate. he did prepare the zip file, now need to pass it in production (after weekly release)
  • 7 March 2025:
    • infra.ci.jenkins.io’s Netlify token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] Netlify token expires the 09 December 2024 · Issue #4446 · jenkins-infra/helpdesk · GitHub)
    • infra.ci.jenkins.io’s NPM token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] NPM token expires the 7 December 2024 · Issue #4447 · jenkins-infra/helpdesk · GitHub)
  • 2026: Code signing Certificate will expire. We’ll have to start the renewal process sooned than last time.
    • Let’s check in September
    • Need payment
    • Biggest target: Windows users (but not only them)
• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • November: $4,3k (invoice)
  • December: $4,4k (invoice)
  • January: $4.3k (invoice)
  • February: $3.4k (forecast at 4k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $8.222 until May 2025

  • November: $13k
  • December: $9,5k
  • January: $13,1k
  • February: $11.2k
    • ci.jenkins.io moved to AWS should decrease costs
    • We did not check and removed manually all of the leftovers though!
      Last 5 days consumption is $835 so about 5k/month and it should decrease in march with the cleanup

• DigitalOcean - Remaining $14.8k until January 02, 2026

  • October: $195.67 (invoice)
  • November: $146 (invoice)
  • December: $192 (invoice)
  • January: $219 (invoice)
  • February: $197
    • Expecting increased bandwidth due to Cloudflare outage 2 weeks ago (BW: $89.16)

• AWS:

  • CloudBees:
    • November: $3,9k
    • December: $540
    • January: $543
    • February: $457 (forecast at $532)
  • Sponsored account (~$57k credits lefts on 31 January 2025, ~60k left until 01/31/2027)
    • November: $482
    • December: $595
    • January: $1.4k
    • February: $6.8k (forecast at $8.7k)

Notes

• Done:

  • Support:

    • docker credentials not up to date
    • [ci.jenkins.io] Paging file too small reported for config-file-provider and email-ext plugin builds
    • [ci.jenkins.io] Docker image build fails consistently in workflow-api-plugin and artifact-manager-s3 tests
    • Keep receiving failure notice while I haven’t done anything for repo
    • [publick8s] Migrate LDAP backup PVC (azureFile) to a CSI Azure File
    • Maven repository settings prevent to download newer versions from Maven Central
    • Instability of artifact-caching-proxy but then GitHub · Where software is built

  • Keep platform up to date:

    • Upgrade to Kubernetes 1.30
    • [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expire February 25 2025

  • ci.jenkins.io to AWS:

    • [ci.jenkins.io] Set up an ECR pull through cache
    • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
    • 5 tests of workflow-durable-task-plugin fail on ci.jenkins.io Windows agents
    • 1 test of sse-gateway-plugin fails on ci.jenkins.io Windows agents

• Closed as not planned:

  • Permissions for anka-build-plugin
  • Password Issue

• Work in Progress:

  • Support:

    • docker-ssh-agent builds consistently timeout on ci.jenkins.io
    • [CD / GitHub Actions] Reached Docker.io rate limit
      • CD release of pipeline-maven failed with 403
    • Instability of artifact-caching-proxy on AWS
    • [pipeline-library/updatecli] Allow custom version and execution in current “pipeline node” context
    • Deploy jenkins-prototype on Netlify
    • New repo scoop-bucket for app manifest distribution
    • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

  • Keep platform up to date:

    • Update-center / crawler root certificate expires the 03th of April 2025 (failing job the 3th of March 2025)

  • Reduce Artifactory storage and bandwidth use:

    • [repo.jenkins-ci.org] Ensure repository incrementals is automatically garbage-collected
    • Reduce Artifactory storage and bandwidth use
    • [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM with Job Cacher

  • Delayed:

    • [ci.jenkins.io] Keep Windows Container Agents or embrace Windows VM agents
    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
    • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• New Issues (to triage):

  • Add a real-world job to weekly.ci.jenkins.io
  • Not able to login to artifactory
  • New mirror in India
  • Upgrade to Kubernetes 1.31
  • Move collection of stats out from Kohsuke’s home
  • Support [skip ci] on default branch
    • Needs discussion on the developer mailing list
    • Delay until March, specific use case, leave
  • Create build for jenkinsci/winp on release ci server
    • Keep it in triage, Alex Earl looking at different paths
    • Needs more discussion about placing code signing certificat inside GitHub action
    • Needs review with Jenkins Security team and Jenkins board
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • build failure with useArtifactCachingProxy=true and dependency with version range
  • External user struggling to submit story to stories.jenkins.io
  • Add monitoring for CD secrets updates
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)