Infrastructure Team Meeting - September 10, 2024

[dduportal (Damien Duportal)]

2024 09 10 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases:
   • Last Week: 2.475 succesfully released Wednesday 4 Sept. (instead of Tuesday 3 as initially planned).
   • This Week: 2.476 started in time
     • Also an important one: ship a fix for a few plugins related to Spring Security
2. Jenkins LTS 2.462.2 succesfully released Wednesday 4 Sept.
   • LTS issue not created as part of the process: gotta check. But delivered on Jenkins Infra. anyway.
3. Next LTS baseline will be selected 18 Sep 2024
   • Will include Spring Security 6 Update (2.475 or later)
   • Will requires JDK17
4. Suspicious PRs last weekend
   • It was a penetration test
   • Remediation actions:
     • Website previews are disabled
   • Immediate actions:
     • Let’s get rid of jenkins4eval (use GHCR instead if needed)
     • ci.jenkins.io: remove credentials for Docker
     • infra.ci:
       • Split Website jobs (and credentials) PR from production (e.g. main branch) deployments
       • Apply Kubernetes PSA to our container agents
       • Split namespace + nodepools + subnets for agents
   • Medium term actions
     • Shared Library for websites or at least readTrusted() / OR find a non infra.ci way to deploy preview (Netlify or back to ci.jio + incremental-like deploy webservice)

Upcoming Calendar

• Next Weekly: 2024-09-17 - 2.477
• Next LTS: 2024-10-30 (Spring Security 6, Java 17 & 21) - Mark Waite release lead
  • Next baseline selection - 2024-09-18
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • Packer credential expired
    • => Issue to create => @jay to work on it
  • 2024-09-19: stats.jenkins.io - New end date for `stats.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: "2024-09-19T23:00:00Z") by jenkins-infra-updatecli[bot] · Pull Request #813 · jenkins-infra/azure · GitHub
    • => Issue to create
  • 2024-09-22: updates.jenkins.io (httpd - redirections) - New end date for `updates.jenkins.io (redirections)` File Share service principal writer on `trusted.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #818 · jenkins-infra/azure · GitHub
    • => Issue to create
  • 2024-09-22: updates.jenkins.io (mirrorbits - content) - New end date for `updates.jenkins.io (content)` File Share service principal writer on `trusted.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #817 · jenkins-infra/azure · GitHub
    • Issue to create
  • 2024-09-22: contributors.jenkins.io - New end date for `contributors.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: 2024-09-22T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #816 · jenkins-infra/azure · GitHub
    • Issue to create
  • 2024-09-26: infra.ci.jenkins.io Azure VM Agents - Extend Azure AD Application password validity on `infra.ci.jenkins.io` (current end date: 2024-09-26T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #822 · jenkins-infra/azure · GitHub
    • Issue to create
  • 2024-10-01: cloudflare API tokens - (private) https://github.com/jenkins-infra/terraform-states/pull/27
    • Issue to create
• Next major event:
  • Adoptium Summit, September 10 (online)
    • Thanks to Bruno Verachten for presenting
  • DevOps World Virtual online September 17, 2024 (online)
    • Jenkins officers and board presentation and Q&A
  • CD Mini Summit in Vienna, September 19, 2024 (on site)
    • Thanks to Olivier Vernin for leading
    • Thanks to Bruno Verachten for presenting

Cloud Budgets

• Azure (CDF paid)

  • June: $4,287 (invoice)
  • July: $4,571 (invoice)
  • August: $4,552 ($4,452 cost + $100 monthly support)
  • September: $1143 consumed (Forecast at ~ $3.4k )
    • LDAP to arm64, 1 less VM! (Migration left over from publicK8s to arm64 · Issue #3837 · jenkins-infra/helpdesk · GitHub)
    • migrate privatek8s to sponsored account → [privatek8s] Migrate AKS cluster to the sponsored subscription · Issue #4250 · jenkins-infra/helpdesk · GitHub

• Azure Sponsorship (Microsoft Credits) - Remaining: $54076 ($45924 consumed) until May 2025

  • June: $7.3k consumed
  • July: $10k consumed
  • August: $10.5k consumed
  • September: $3364 (Forecast at ~10k)

• DigitalOcean - Remaining ~$15,926 (~4k consumed) until 02 January 2025

  • June: $165.32 (invoice)
  • July: $176.01 (invoice)
  • August $200.08 (invoice)
    • Bandwidth increase (archives.jenkins.io: fallback of get.jenkins.io)
  • September: $52 (Forecast at $160)

• AWS:

  • CloudBees:
    • June: $5,862
    • July: $6.5k
    • August: $6.3k
    • September: $2118, forecast at 6.5k
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

• to open an issue describing the “cloud billing plans” for the upcomings 4 months

Notes

• Done:

  • Suspicious actions in jenkinsci and jenkins-infra
  • [updatecli][2024-09] YAML cleanups and tracking missing dependencies
  • Postpone 2.475 weekly release from 9/3 to 9/4
  • Not able to create account

• Closed as not planned:

  • Changing owners of LoadNinja Jenkins Plugin
  • loadninja_devops
  • I’m not receiving any recovery emails

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • We succesfully ran 2 brownouts of one hour each \o/
    • We discovered a few MINOR issues=> fixed
      • Forgot PagerDuty
      • Forgot updates.jenkins-ci.org domain
      • Forgot to keep a DNS record on the former service (even if not used) to keep update_center2 working during brownouts
    • We listed (and are working) on improvement of our tooling for the new service (monitoring, cleanups)
    • We are ready for a 24 hours brownout:
      • Proposal: Wednesday 14h00 UTC until Thursday
        • Metrics
        • Error logs
        • Billing
    • Fun logs during the brownout: there are Artifactory instances pointing at UC
    • Next week: final GO/NO GO
  • Add JDK21 agents (build)
    • Packer Image VM template version 2.0.0 features “No more default JDK in PATH”
    • Next step is to setup Windows SSH agents to use 2.0.0 and specify JDK11/17/21 paths
    • Blocked by Packer expired credential (cannot deliver 2.0.0)
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • Cron a geoip updater task for mirrorbits
      • Work in progress on the custom Docker image (geoIpupdater + azcopy)
      • Then helm chart
      • And then deploy it!
  • GSoC:
    • To host stats.jenkins.io GSoC 2024 project in jenkins-infra
      • Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io
        • Proposal: DNS migration Wednesday 11 Sept.
        • Proposal: 1 week before disabling GH page old site (and rename the branch to ‘data’)
    • Multiple requests for GSoC 2024 Plugin Modernizer Tool
    • Adding GSoC project to jenkins-infra
    • RPU GSoC project with terraform integration on infra’s backend
  • [oic-auth-plugin] CodeCov stopped working 1 month ago
  • Gradle plugin uses a proprietary dependency
  • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit

• ToDo (next milestone) (infra-team-sync-2024-09-17 Milestone · GitHub)

  • Ensure our build agents have enought entropy (rngd, haveged, etc.)
  • Ask OSSPlanet for a Jenkins mirror