Infrastructure Team Meeting - December 03, 2024

[dduportal (Damien Duportal)]

2024 12 03 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.487 was released succesfully as part of the Security Advisory Wednesday 27 Nov.
   • This Week: 2.488 started on time should be finished later today
2. Jenkins Election results have been announced: 2024 Jenkins Board and Officer Election Results
   • Few minor optimizations on the blog post to run but the essential is there

Upcoming Calendar

• Next Weekly: 2.489, 10 Dec. 2024
• Next LTS: 2.479.3, 8 Jan. 2025
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 09/10 Dec. Netlify tokens => 2 tokens to be renews
  • Terraform Azure:
    • Azure File Share Principal `updates.jenkins.io (redirections)` on `trusted.ci.jenkins.io` expires on `2024-12-18T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #886 · jenkins-infra/azure · GitHub => to be closed (as we use NFS + Rsync now)
    • Azure File Share Principal `updates.jenkins.io (content)` on `trusted.ci.jenkins.io` expires on `2024-12-18T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #885 · jenkins-infra/azure · GitHub => to be closed (as we use NFS + Rsync now)
    • 18 Dec.: stats.jenkins.io, contributors.jio and infra.ci.jio
• Next major event:
  • 2 week “release break” end of year
  • Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025

Cloud Budgets

• Azure CDF:
  • September: $3,810 (invoice)
  • October: $4.0k (invoice)
  • November: $4.3k
    • New Update Center, impact on the Azure outbound bandwidth (slight increase: from ~$30 daily to ~$45 daily)
  • December: $328 (too soon for forecast)
• Azure Sponsorship (Microsoft Credits) - Remaining: $40952 ($79048 consumed) until May 2025
  • September: $10.4k consumed
  • October: $12.9k consumed
  • November: $13k
  • December: $635 (too soon for forecast)
• DigitalOcean - Remaining $15,478 ($4k consumed) until 02 January 2025
  • September: $158 (invoice)
  • October: $195.67 (invoice)
  • November: $146 (Forecast at $175)
  • December: $12 (too soon for forecast)
    • Requesting extension
• AWS:
  • CloudBees:
    • August: $6.3k
    • September: $6.1k
    • October: $6.4k
    • November: $3.9k
    • December: $42 (too soon for forecast)
  • Sponsored account
    • October: $178
    • November: $482
    • December: $35 (too soon for forecast)

Notes

• Done:

  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart

    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
    • Database updated by a cronjob, once a day (for now)
    • Minor cleanup + 24h → 72h interval to set in a few days if no issues

  • New Update Center

    • HTTP errors on multiple sites
    • Wrong update-center returned for 2.452.4
    • [Update Center] “eventually consistent” Azure Shared File storage for HTTPD htdocs (aka. corrupted files some time to time)

  • Keeping the Infra. Up to Date:

    • [updatecli] Track GitHub SSH in bound IPs allowed from our controllers and agents
      • Our agents can only emits SSH request to GitHub (instead of full internet)
    • Remove BlueOcean from all of our controllers
      • All controllers are clean from BlueOcean
      • New issue to replace Stage View by pipeline graph history
    • [infra.ci.jenkins.io/packer-images] Azure Credential used by Packer Image builds expire december 9 2024
      • Jay discovered the whole process with the help of Stéphane \o/ Damien not a SPOF for this.
    • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.479.2

  • Level 1 Support:

    • Rename freestyle-project to freestyle-project-plugin
    • Unable to create a new account
    • Jira and github account linkage
    • CI/CD check stuck in jenkins-io-components PR #150
    • [infra-admin] Renew VPN certificat for smerle as it expired today
    • CD secrets appear to have expired
    • Request to be made a maintainer of the jenkins-infra/jenkins-io-components repo

  • Spam users:

    • Block user philpcstart due to Jenkins issue spam
    • Let’s stop reporting all accounts we block on helpdesk: we know we have quite a few. Helpdesk issue can be used by non admin to ask for blocking spam user though

  • ci.jenkins.io to AWS:

    • (packer-images): Add Garbage Collector for AWS
      • Works very well \o/
    • [ci.jenkins.io] Define virtual networking for AWS

• Closed as not planned:

  • Assistance Needed to Recover Corporate Jenkins Account

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • [Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas
      • Nothing done, todo
  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • Working on network routing (blocking adding nodes)
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • Working on network routing (blocking adding nodes)
    • [ci.jenkins.io] Move controller (VM) to AWS
      • On Hold (waiting for agents setups)
  • build failure with useArtifactCachingProxy=true and dependency with version range
    • Legit request BUT it looks like it is the only case in jenkinsci (or one of a few)
    • Let’s add something on the dev. documentation to advertise not to use version range
    • ACP should not fail: there is something to be diagnosed
  • External user struggling to submit story to stories.jenkins.io
    • Nothing done, low priority
  • Infra stats missing October 2024 data for stats.jenkins.io Plugin Installation Trend feature
    • Waiting for KK
  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org
    • Nothing done, low priority
    • Only a (?small) “subset” of artefacts
  • Add monitoring for CD secrets updates
    • Jay is working on shared library (knowledge sharing and learning)

• New Issues (to triage/staying in triage zone):

  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • [ci.jenkins.io] Move ephemeral Linux containers to AWS
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2024-12-10 Milestone · GitHub)

  • [Update Center] HTTP/404 on /current/updates/*.json* links
    • Edge case, but worth fixing it in crawler publication script
  • dnf5 update fails with gpgcheck=1
    • User has a solution provided (in www.jenkins.io)
    • pkg.jenkins.io pages need an update to point to www.jenkins.io installation
  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • Stephane new (minor) issues
    • cleaning and changing the cron for geoip updater cronjob · Issue #4436 · jenkins-infra/helpdesk · GitHub
    • change the stage view to pipeline graph view on infra.ci and then ci.jio · Issue #4435 · jenkins-infra/helpdesk · GitHub