Infrastructure Team Meeting - November 26, 2024

[dduportal (Damien Duportal)]

2024 11 26 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases: none today (2.487 part of the security advisory on core tomorrow)
2. Git LFS 3.6.0 on Docker images (CVEs)
   • Controller was failing (unavailable package on some CPU archs). Thanks mark for the fix!
   • Many different versions for git-lfs:
     • Debian (fix by Mark) with tar.gz
     • RedHat with package manager (UBI)
     • Alpine (with package manager, no tar gz)
   • docker(.*)-agent images are using non pinned version!
3. Security Advisory tomorrow (https://groups.google.com/g/jenkinsci-advisories/c/KPuGGzCkWX0)
   • Weekly 2.487
   • LTS 2.479.2
   • Plugins

Upcoming Calendar

• Next Weekly:
  • 2.487: See announcements
  • 2.488: 3 Dec. 2024
• Next LTS: 2.479.3, 8 Jan. 2025
• Next Security Release as per jenkinsci-advisories: Yes (see announcements) https://groups.google.com/g/jenkinsci-advisories/c/KPuGGzCkWX0
• Upcoming credentials expirations (~3 weeks):
  • Packer credentials in infra.ci (09 Dec. 2024) - Extend Azure AD Application password validity for packer-resources (current end date: 2024-12-09T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #874 · jenkins-infra/azure · GitHub => @smerle
  • Netlify tokens (Damien received an email) => 2 tokens to be renews
• Next major event:
  • Jenkins Elections - voting started 1st of November, registration reopened, over 50 votes already cast from 78 registered voters
  • 2 weeks “Releases break” at the end of calendar year
    • Delays LTS 2.479.3 to 8 Jan 2025
  • Jenkins Contributor summit - 31 January 2025 in Brussels, Belgium
  • FOSDEM - 1 and 2 February - Brussels, Belgium

Cloud Budgets

• Azure CDF:
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $4.0k (invoice)
  • November: $3.5k (forecast at $4.4k)
    • New Update Center, impact on the Azure outbound bandwidth (slight increase: from ~$30 daily to ~$45 daily)
• Azure Sponsorship (Microsoft Credits) - Remaining: $43891 ($76109 consumed) until May 2025
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $12.9k consumed
  • November: $10k (Forecast at $11k)
• DigitalOcean - Remaining $15,478 ($4k consumed) until 02 January 2025
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $195.67 (invoice)
  • November: $146 (Forecast at $175)
    • Requesting extension in Dec.
• AWS:
  • CloudBees:
    • August: $6.3k
    • September: $6.1k
    • October: $6.4k
    • November: $3.8k (forecast at $4.3k)
      • New Update Center: daily billing dropped from ~200 (last 3 months) to ~16 (last 7 days)!
  • Sponsored account
    • October: $178
    • November: $350 (forecast at ~$450)

Notes

• Done:

  • Spam users:

    • Block user carolgrey98 for Jira issue spam
    • Block user tasha981 due to Jira issue spam
    • Block user tasha981 for Jira issue spam
    • Block user christin602bey for Jira issue spam
    • Block user jane3144fitzgibbon for nonsense Jira issue comment
    • Block user james598goff for Jira issue spam
    • Block user jody998 for Jira issue spam
    • Block user julie598doyle for Jira issue spam
    • Block robert344humphries due to Jira issue spam
    • Block user ruby05guy due to Jira issue spam
    • Block user judith75hodges due to Jira issue spam
    • Block user velma41abbott for Jira issue spam
    • Block james598keen user due to spam comment in Jira issue
    • Block merry678 user due to Jira issue spam

  • 401 error when publishing new plugin version

    • Caused by work on (AWS) ci.jenkins.io ephemeral agent configuration which broke Azure VM SSH agent allocation

  • New Update Center:

    • 404 downloading some tool installer JSONs (geo dependant)
    • Base container jenkins servers - Plugins 404 errors

  • Keeping the Infra. Up to Date:

    • [cert.ci.jenkins.io] Keep Maven tools up to date with updatecli
    • [packer-images] track Linux base images with updatecli

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • We discovered the hard way that BIC on Cloudflare was still enabled onus east mirror. Fixed
    • [Update Center] “eventually consistent” Azure Shared File storage for HTTPD htdocs (aka. corrupted files some time to time)
      • WiP on replacing azcopy by rsync (from update-center2)
      • Then httpd + rsync side svc
    • [Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas
      • ToDo

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • [ci.jenkins.io] Define virtual networking for AWS
    • [ci.jenkins.io] Move controller (VM) to AWS
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • (packer-images): Add Garbage Collector for AWS
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • Add API IP restriction and then go

  • Keeping the Infra. Up to Date:

    • Remove BlueOcean from all of our controllers
      • infra.ci, release.ci and weekly.ci done
      • Next: ci.jio! let’s proceed
    • [updatecli] Track GitHub SSH in bound IPs allowed from our controllers and agents
    • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
      • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits

  • Level 1 Support:

    • External user struggling to submit story to stories.jenkins.io
    • Infra stats missing October 2024 data for stats.jenkins.io Plugin Installation Trend feature
    • Maven Central artifacts are being downloaded from repo.jenkins-ci.org
    • plugin stats for bouncycastle-api are missing data.
      • Not infra problem but tool issue of the stats data
      • Let’s remove from milestone as no action expected (or posible) by jenkins-infra team

• New Issues (to triaged):

  • Request to be made a maintainer of the jenkins-infa/jenkins-io-components repo
  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • [ci.jenkins.io] Move ephemeral Linux containers to AWS
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2024-12-03 Milestone · GitHub)