Infrastructure Team Meeting - September 03, 2024

[dduportal (Damien Duportal)]

2024 09 03 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)
• @jayfranco999 (Jay Reddy)

Announcements

1. Weekly Releases
   • 2.474 succeeded. No package job issue (e.g. OSUOSL wasn’t slow)
   • 2.475
     • Delayed of 1 day (from today to tomorrow, e.g. 2024-09-04) after the LTS as per Postpone 2.475 weekly release from 9/3 to 9/4 · Issue #4274 · jenkins-infra/helpdesk · GitHub
     • will have Jetty 12 EE 9 and Spring Security 6.x.
2. Looks like some of us have issues with the infra SIG Google Agenda event => need to check
3. Tomorrow (2024-09-04): LTS then weekly

Upcoming Calendar

• Next Weekly: 2024-09-10 - 2.476
• Next LTS: 2024-09-04 (tomorrow) - 2.462.2 - Alex Brandes is release lead
  • Next baseline selection - 2024-09-18
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2024-09-19: stats.jenkins.io - New end date for `stats.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2024-09-19T23:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #813 · jenkins-infra/azure · GitHub
    • => Issue to create
  • 2024-09-22: updates.jenkins.io (httpd - redirections) - New end date for `updates.jenkins.io (redirections)` File Share service principal writer on `trusted.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #818 · jenkins-infra/azure · GitHub
    • => Issue to create
  • 2024-09-22: updates.jenkins.io (mirrorbits - content) - New end date for `updates.jenkins.io (content)` File Share service principal writer on `trusted.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #817 · jenkins-infra/azure · GitHub
    • Issue to create
  • 2024-09-22: contributors.jenkins.io - New end date for `contributors.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #816 · jenkins-infra/azure · GitHub
    • Issue to create
• Next major event:
  • Adoptium Summit, September 10 (online)
    • Thanks to Bruno Verachten for presenting
  • DevOps World Virtual online September 17, 2024 (online)
    • Jenkins officers and board presentation and Q&A
  • CD Mini Summit in Vienna, September 19, 2024 (on site)
    • Thanks to Olivier Vernin for leading
    • Thanks to Bruno Verachten for presenting

Cloud Budgets

• Azure (CDF paid)

  • June: $4,287 (invoice)
  • July: $4,571 (invoice)
  • August: $4,552 ($4,452 cost + $100 monthly support)
  • September: $266 consumed (Forecast at ~ $3.5k )
    • Redis instances removed from this account: should decrease the bill of 8-10%
    • LDAP to arm64, 1 less VM! (Migration left over from publicK8s to arm64 · Issue #3837 · jenkins-infra/helpdesk · GitHub)
    • migrate privatek8s to sponsored account → [privatek8s] Migrate AKS cluster to the sponsored subscription · Issue #4250 · jenkins-infra/helpdesk · GitHub

• Azure Sponsorship (Microsoft Credits) - Remaining: $56624 ($43376 consumed) until May 2025

  • June: $7.3k consumed
  • July: $10k consumed
  • August: $10.5k consumed
  • September: $815 (Forecast at ~8k)

• DigitalOcean - Remaining ~$15,967 (~4k consumed) until 02 January 2025

  • June: $165.32 (invoice)
  • July: $176.01 (invoice)
  • August $200.08 (invoice)
    • Bandwidth increase (archives.jenkins.io: fallback of get.jenkins.io)
  • September: $11 (Forecast at $120)

• AWS:

  • CloudBees:
    • June: $5,862
    • July: $6.5k
    • August: $6.3k
    • September: $530, forecast at 6.3k
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

• to open an issue describing the “cloud billing plans” for the upcomings 4 months

Notes

• Done:

  • ACP mirroring configuration prevents usage of external repositories
  • Block user santaender for issue spam
  • Block user rhythmtutu for issue spam
  • [private.vpn.jenkins.io] 2024-09-06 (September 2024) VPN CRL expires
    • Another PR to “automate” opening issues so we don’t rely on calendar
    • Long term: automate this rotation (requires Azure Vault to host the admin certificate credential)
  • [get.jenkins.io, mirrors.updates.jenkins.io] Optimize Redis used by mirrorbits instances (costs, security and performances)
    • Lesson: migrating Redis data for mirrorbits is useless and slow. Next time we’ll rely on fallbacks from migration.

• Closed as not planned:

  • unexpected plugin versions being returned by update-center

• Work in Progress (Milestone 125

  • Postpone 2.475 weekly release from 9/3 to 9/4
  • Requesting VPN access for admin privileges on Jenkins-infra
    • @dduportal to open an issue for improving infra.ci matrix permission
  • Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io
    • @lemeurherve to prepare a plan for production
    • Don’t forget about credential for file storage
    • Don’t forget about blue/green fallback scenario
    • TTL for the plan: 17 Sept.
    • Last mile of To host stats.jenkins.io GSoC 2024 project in jenkins-infra
  • Gradle plugin uses a proprietary dependency
  • Multiple requests for GSoC 2024 Plugin Modernizer Tool
    • Need to migrate repo and workflows to jenkins-infra + infra.ci with reports.jio
  • Adding GSoC project to jenkins-infra
    • Need to migrate GH repo to jenkins-infra
  • RPU GSoC project with terraform integration on infra’s backend
    • Need planning inside the infra team
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • We want to handle GeoIP database updates without hitting Rate limit or Azure Fileshare SMB errors
    • We are going for a Kubernetes CronJob (need helm chart) in the publick8s cluster (close to mirrorbits instances)
    • With a custom Docker image for this implementation (need geoipupdate CLI and also azcopy CLI)
  • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
    • Fixed the JobDSL “whitespace” issues which were breaking infra.ci following latest @dduportal changes
    • WiP on the JobDSL organization folder implementation
    • Next step: pipeline shared library changes for the way we publish Docker images
  • Add JDK21 agents (build)
    • WiP on JDK17/21 Windows for cert.ci and trusted.ci (Windows SSH agents)
      • solution validated and being implemented in packer-image templates
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Redis migration done: ready to production
    • We have to plan brownout:
      • 1 hour this Thursday 05 Sep. (Morning in EU, after the LTS/weekly release)
      • 1 hour this Monday 09 Sep. (Morning in US)
      • We have to check, before brownout, how to detect errors (datadog)
    • In parallel (but lower priority): use mirrorbits CLI on trustedci (instead of kubectl)
      • Excuse to validate use of Azure Private Link Services for Kubernetes internal LBs

• ToDo (next milestone) (https://github.com/jenkins-infra/helpdesk/milestone/[ID+1])

  • account issue: Not able to create account · Issue #4270 · jenkins-infra/helpdesk · GitHub
  • Codecov/GHA support question for plugin maintainer: [oic-auth-plugin] CodeCov stopped working 1 month ago · Issue #4267 · jenkins-infra/helpdesk · GitHub