Infrastructure Team Meeting - February 11, 2025

[dduportal (Damien Duportal)]

2025 02 11 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)

Announcements

1. Jenkins Weekly Releases
   • Last weeks:
     • 2.495 released with no issues the 28 January 2025 - You're invited to talk on Matrix
     • 2.496 released with no issues the 4 February 2025 - You're invited to talk on Matrix
   • This Week: 2.497 started on time and should be released and deployed later today - You're invited to talk on Matrix
   • LTS release last week (5 Feb. 2025) 2.492.1
2. Outages:
   • Azure Container Instances: no more outage, back to normal
   • Cloudflare R2 outage the 6 February 2025 (see Cloudflare Status - Cloudflare R2 Availability Issues)
     • No impact for our users: Update Center requests were redirected to archives.jenkins.io or served without problem (except slightly increased latency) by R2 HTTP endpoints
     • Update center metadata were only updated on archives during the incident though
   • LDAP Outage Friday 7 February 2025: described in LDAP server is down, login fails to repo.jenkins-ci.org and accounts.jenkins.io · Issue #4531 · jenkins-infra/helpdesk · GitHub
     • Root cause: sudden disk issues due to missing Azure permissions. Fixed
     • Impacted LDAP but also weekly.ci, infra.ci and release.ci
3. Migration of ci.jenkins.io to AWS scheduled this Thursday 13 February 2025 at 09:00UTC
4. Damien off from 24 February up to 3 March included
   • Need someone to lead the infra weekly meeting the 25 February
     • @smerle volunteers
   • Need someone to watch the Weekly release the 25 February
     • @smerle volunteers

Upcoming Calendar

• Next Weekly: 2.498, 18 February
• Next LTS: 2.492.2, 5 March, Release lead is Kris Stern
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 16 February 2025: Azure SP for Release Azure Vault
    • Damien might be a SPOF: need to run it in pair with Stephane to ensure it is not the case
  • 25 February: Azure SP used by Packer expires
    • Issue to create => @smerle
    • PR: Extend Azure AD Application password validity for packer-resources (current end date: 2025-02-25T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #929 · jenkins-infra/azure · GitHub
  • 3 March 2025: Update Center Signing Certificate expires
    • Need an issue (Previous one was Update-center / crawler root certificate expires the 04 of May (failing job 1 month before) · Issue #4026 · jenkins-infra/helpdesk · GitHub)
    • Damien might be a SPOF: need to run it in pair with Stephane to ensure it is not the case (or it is acceptable)
  • 7 March 2025:
    • infra.ci.jenkins.io’s Netlify token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] Netlify token expires the 09 December 2024 · Issue #4446 · jenkins-infra/helpdesk · GitHub)
    • infra.ci.jenkins.io’s NPM token expires
      • Need an issue (previous one was [infra.ci.jenkins.io] NPM token expires the 7 December 2024 · Issue #4447 · jenkins-infra/helpdesk · GitHub)
• Next major event: N.A.

Cloud Budgets

• Azure CDF:

  • November: $4,3k (invoice)
  • December: $4,4k (invoice)
  • January: $4.3k (invoice)
  • February: $1.5k (forecast at 4k)
    • Forecast is pessimistic due to the cost incurred by Azure outage on our AKS clusters
    • Increase in DNS costs (public requests, mostly due to Cloudflare outage)

• Azure Sponsorship (Microsoft Credits) - Remaining: $12.750 until May 2025

  • November: $13k
  • December: $9,5k
  • January: $13,1k
    • Bom, bom, bom
  • Feburary: $6.1k (forecast at $11,8k)
    • we MUST move to AWS + delay BOM builds while on Az

• DigitalOcean - Remaining $14.9k until January 02, 2026

  • November: $146 (invoice)
  • December: $192 (invoice)
  • January: $219 (invoice)
  • Feburary: $136 (forecast at $212)

• AWS:

  • CloudBees:
    • November: $3,9k
    • December: $540
    • January: $543
    • February: $323 (forecast at $531)
  • Sponsored account (~$55k credits lefts)
    • November: $482
    • December: $595
    • January: $1.4k
    • February: $733 (forecast at $1.5k, even more once migration of ci.jio is complete)

Notes

• Done:

  • Contributions
    • Broken “thanks” message on plugins.jenkins.io
    • [INFRA-2178] No size limit in Last name
  • Support
    • [Incident] Windows build of plugins don’t start on ci.jenkins.io
    • LDAP server is down, login fails to repo.jenkins-ci.org and accounts.jenkins.io
    • Request to Change Username or Delete Account
    • jenkinsci-jam mailing list accepts and distributes spam
    • I lost my credentials
  • Keep platform up to date
    • SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.492.1
    • JDK patch upgrade campaign (January 2025)
    • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2025-02-06
    • [private.vpn.jenkins.io] 2025-02-23 (February 2025) VPN CRL expires
    • [infra.ci.jenkins.io] 2025-02-18 Expiration of the Digital Oceans PATs used by Terraform
    • [trusted.ci.jenkins.io] 2025-02-16 Credential for spawning Azure VM agents expires
    • [terraform-aws-sponsorship] updatecli manifest to track karpenter helm_release version
      • Led to and updatecli bug report (related to helm chart source / credential for OCI repositories)
    • [terraform-aws-sponsorship] updatecli manifest to track AWS Load Balancer helm_release version
    • chore bug(openvpn/CRL): updatecli to open issue for CRL renewal is not working
      • Hacking until the feature is in updatecli
  • ci.jenkins.io to AWS
    • [ci.jenkins.io] Move ephemeral VM agents to AWS

• Closed as not planned:

  • I could able to login in my account
  • pkg keyring expired
  • Help me create a user id in Jenkins
  • hii
  • Failure while running plugin BOM processes

• Work in Progress:

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • Migration 13 Feb.: need to announce it (mailing list + Element message)
      • We all agree on not needing a blog post
    • Ping Mark (bom team) once done once we’ll kick the “bom” controller build on new ci.jio
    • [ci.jenkins.io] Set up an ECR pull through cache
    • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
    • [ci.jenkins.io] Move controller (VM) to AWS

  • Reduce Artifactory storage and bandwidth use

    • Darin leads the effort. We already see storage decrease thanks to his efforts
    • [repo.jenkins-ci.org] Atlassian Mirror Repository: Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/
    • [INFRA-2821] Remove unused incrementals
    • Foreign releases in public repository

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Decommision config from ‘pkg’ VM => was delayed due to FOSDEM
    • Now we have 2 LTS lines since the “update center HTTPS fix”, we’ll have to schedule enforcing TLS on Update Center => target in March
      • Blog post may be written before

  • Support

    • [trusted.ci.jenkins.io] Crawler fails to publish new tools metadata due to an S3 ↔ Cloudflare R2 error s3:PutObject NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER not implemented
      • Change of plan: we discarded the “Install 2 AWS CLI with a pinned old version for Cloudflare”
      • Rationale: Cloudflare support (in their discourse forum in which we posted) gave us a new flag for the AWS CLI to use, making the latest version working perfectly. Crawler is using it with success
      • Scope of the issue is extended: we have to apply the fix to Update Center and then make sure it uses the latest AWS CLI (as we discovered we forgot to keep it updated)
    • Instability of artifact-caching-proxy
      • Only concern is for BOM build
      • Short term: we want to add client-side cache ($HOME/.m2/repository) for BOM only
        • Job Cacher plugin, or custom caching technique with PVC in pod agents
      • Delayed until AWS migration
    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
      • No news from Andrew, lets wait
    • Deploy jenkins-prototype on Netlify
      • Delayed (ready to work on but not in the top priority list)
    • Support [skip ci] on default branch
      • Delayed (ready to work on but not in the top priority list)
    • ci.jenkins.io: upgrade datadog plugin from 8.x to 9.x
      • Required before migration to AWS
    • Create build for jenkinsci/winp on release ci server
      • Delayed (ready to work on but not in the top priority list)
    • New repo scoop-bucket for app manifest distribution
      • Delayed (ready to work on but not in the top priority list)

  • Keep platform up to date

    • [release.ci.jenkins.io] 2025-02-16 Azure Credentials for Core Release (Vault access) expires

  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)

    • Add monitoring for CD secrets updates

• New Issues (to triage):

  • build failure with useArtifactCachingProxy=true and dependency with version range
  • External user struggling to submit story to stories.jenkins.io
  • [Update Center] HTTP/404 on /current/updates/*.json* links
  • dnf5 update fails with gpgcheck=1
  • Move collection of stats out from Kohsuke’s home
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)

• ToDo (next milestone) (infra-team-sync-2025-02-18 Milestone · GitHub)

  • Upgrade to Kubernetes 1.30
    • @smerle leads
    • One PVC (LDAP backup on Azure file) to migrate to CSI before upgrading publick8s => @dduportal
  • [ci.jenkins.io] Enable Maven dependencies client-side caching for BOM with Job Cacher
  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org
    • Might (or might NOT) be related to Darin’s efforts
  • X (formerly twitter) account looks obsolete (when it is not)
    • Fowarding to Developer Evangelist
  • LTS Changelog archive page only displays changes since baseline
    • Move to jenkins.io issue tracker (not infra)
    • We have to check that the contributor keeps an eye and propose a fix (or we revert their change if we are sure it is the cause)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
    • Started on FOSDEM => @dduportal to write report on the issue