Infrastructure Team Meeting - May 21, 2024

[dduportal (Damien Duportal)]

2024 05 21 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• Jay Reddy

Announcements

1. Weekly
   • 2.459 release is in progress, changelog merged, containers building
     • Packaging job regularly blocks on synchronization step
2. LTS
   • 2.452.1 was released last week
   • All the infra was upgraded between Wednesday and Thursday
3. Welcome Jay!

Upcoming Calendar

• Next Weekly: 2.460, 28 May 2024
• Next LTS:
  • RC 2.452.2, Wed. 29 2024
  • 2.452.2, Jun. 12 2024
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • OSS Summit, Sept. in Vienna (Bruno V. and also our former Olivier Vernin friend!)
  • CD “mini” summit, the day after (same location, same players!)

Cloud Budgets

• Azure (CDF paid)
  • February: $4,147.76 (invoice)
  • March: $4,398 (invoice)
  • April: $4,550 (invoice)
  • May (current): $2828 consumed (Forecast at ~4.2k)
• Azure Sponsorship (Microsoft Credits)
  • Global Status:
    • Credits left: $27,960 until 31 August 2024
  • February: $1,992
  • March: $2,375
  • April: $2,041
  • May (current): $2656 (Forecast at ~4k)
• DigitalOcean
  • Global Status:
    • Credits left: $16,560.58 until January 2025
  • February: $650
  • March: $938
  • April: $840
  • May (current): $606 (Forecast at ~$890)
• AWS:
  • CloudBees:
    • February: $8,557
    • March: $9,567
    • April: $9,782
    • May (current): $6,361 (Forecast at ~$9.4k)
  • Sponsored account
    • Untouched (60k until January 2025)
    • Request submitted for 2025 donation from AWS
      • Mark Waite submitted, answer expected in June/July 2024

Notes

• Done:

  • Block spammer patrick521f
  • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.452.1
  • Grant public visibility on GSoC 2024 - OpenRewrite GitHub project
  • [wiki (docker-confluence-data) deprecated updatecli manifest
  • [infra.ci.jenkins.io / accountapp] Build stuck waiting for jdk17 label

• Closed as not planned:

  • Configuration as code plugin automated release failed upload to repo.jenkins-ci.org
  • retrieve my id and password

• Work In Progress:

  • [ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account

    • Successfully migrated (EKS + DigitalOcean) to Azure. Bills shows it does what we want, and no error on ci.jio
    • Cleanup in progress

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Proposal by @smerle for the production opening: add 1-2 mirrors on the currently unused OSUOSL VMs we have
      • We have to ask OSUOSL => @dduportal (+ ask about the OSUOSL mirrors slowness when syncing)
      • Enough storage? => @smerle to check
      • But we keep the plan of adding a 2nd Cloudflare in US though (to spread workloads) => @smerle
    • WiP: update-center2 to populate the distincts webservices => @dduportal

  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release

    • WiP:
      • Changed the plan: we will define disk, PV and PVC as code (static provisionning of storage, instead of dynamic with ony PVC)
      • weekly.ci is the first target, @smerle will create resources when he’s back Thursday. Plan for real migration to be detailled but looks good (including migration)
        • @dduportal to check the latest (draft) PR
    • Next steps:
      • infra.ci and release.ci

  • Packer-Images: bug within updatecli to update goss since exclusion of windows 2019 for vsstudio

    • “updatecli” process is broken since introduction of VSCode 2019, we wern’t able to release new container image since 4 weeks due to this
    • Root problem identified and required 2 steps:
      • Done: Manifests short term fix, we receieved (and emerged) 38 PRs, image released and deployed
      • WiP: Fixing the goss testing system for VSCode 2019 (by @smerle)

  • packer-images: git version not always available simultaneously on apt ppa for arm64 and amd

    • Another issue which delayed the agent image of a few days: the Git version between Windows, Ubuntu x86_64 and arm64 where ALL different!
      • As such, delivering the fixed CVEs on git is again an issue
    • Version upgraded and delivered
    • WiP: we are trying to switch to compiling git ourselves on Linux
      • Works fast and easy
      • Currently cleaning up + trying to check for explicit dependencies git
      • Tests to add to make this safer for the future
      • Todo: updatecli to update git source version

  • Stuck issues (e.g. no work done but keep in upcoming milestone)

    • Add .war.asc to get.jenkins.io
    • New Jenkins mirror in Romania by RCS&RDS
    • New Jenkins mirror in Romania by Hostico
    • [[Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts](https://
    • Enable 2FA on jenkinsci npm account
    • Replace Blue Ocean in default display URL (or remove the Blue Ocean plugins)
    • [infra.ci.jenkins.io] Abort and remove (immediately) orphaned builds

• ToDo (next milestone) (infra-team-sync-2024-05-28 Milestone · GitHub)

  • Request Temporal “Admin Read” Access for https://ci.jenkins.io/
    • @dduportal to discuss with Carlos
  • ftp.halifax.rwth-aachen.de blocked
    • Closeable with a huge thanks to C-Otto