Infrastructure Team Meeting - August 27, 2024

[dduportal (Damien Duportal)]

2024 08 27 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)
• @jayfranco999 (Jay Reddy)

Announcements

1. Weekly
   • 2.473 succeeded.
     • Package job aborted: still (slow) OSUOSL blocking
   • 2.474 started on time today
2. GitHub spam comments on jenkins-infra GH Org
   • Abuse requests sent by Daniel
   • Widespread: temurin and other projects are also subject to this
   • We can ban user immediately, and hide comments, but no deletion

Upcoming Calendar

• Next Weekly: 2024-09-03 - 2.475
  • will have Jetty 12 EE 9 and Spring Security 6.x. Risk that it might be delayed of 1 week though.
• Next LTS: 2024-09-04 - 2.462.2 - Alex Brandes is release lead
  • Next baseline selection - 2024-09-18
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2024-09-06: VPN CRL expiration - [private.vpn.jenkins.io] `2024-09-06` (September 2024) VPN CRL expires · Issue #4266 · jenkins-infra/helpdesk · GitHub
• Next major event:
  • DevOps World Virtual online September 17, 2024
    • Jenkins officers and board presentation and Q&A
  • CD Mini Summit in Vienna, September 19, 2024
    • Thanks to Olivier Vernin for leading
    • Thanks to Bruno Verachten for presenting

Cloud Budgets

• Azure (CDF paid)

  • May: $4,339 (invoice)

  • June: $4,287 (invoice)

  • July: $4,571 (invoice)

  • August: $3.8k consumed (Forecast at ~4.5k )

    
    • Redis instances: [get.jenkins.io, mirrors.updates.jenkins.io] Optimize Redis used by `mirrorbits` instances (costs, security and performances) · Issue #4259 · jenkins-infra/helpdesk · GitHub
    • mirrorbits runs in arm64, 1 less VM! (Migration left over from publicK8s to arm64 · Issue #3837 · jenkins-infra/helpdesk · GitHub)
    • migrate privatek8s to sponsored account → [privatek8s] Migrate AKS cluster to the sponsored subscription · Issue #4250 · jenkins-infra/helpdesk · GitHub

• Azure Sponsorship (Microsoft Credits) - Remaining: $59624 ($40376 consumed) until May 2025

  • May: $5k consumed
  • June: $7.3k consumed
  • July: $10k consumed
  • August: $8.4 consumed (Forecast at 9.8k/10k)

• DigitalOcean - Remaining ~16k$ (4k consumed) until 02 January 2025

  • May: $648
  • June: $165.32
  • July: $176 consumed
  • August $172.89 (Forecast at $200)
    • Bandwidth increase

• AWS:

  • CloudBees:
    • May: $8,281
    • June: $5,862
    • July: $6.5k
    • August: , forecast at 6.5k
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • Revoke accesses for Herve (temporary)
  • get.jenkins.io cannot geolocate my IP address on 50% of requests
    • Reopened GeoIP update issue
  • Account deletion
  • Credential rotations:
    • [infra.ci.jenkins.io] 2024-09-05 Expiration of the Digital Oceans PATs used by the Terraform project digitalocean
    • [trusted.ci.jenkins.io] 2024-09-02 Credential for spawning Azure VM agents expires
    • [release.ci.jenkins.io] 2024-08-31 Azure Credentials for Core Release (Vault access) expires
  • Bump Maven to 3.9.9
  • Cannot release a new plugin with CD even if tokens are created and automatic PR on RPU merged
  • [infra.ci.jenkins.io] Azure VM agent init script not in sync with Puppet code
    • Caught while working on adding new VM JDKs (see below)
  • [accounts.jenkins.io] Decrease Google indexing
  • [ci.jenkins.io] Bump datadog plugin from 7.0.1 to 7.1.2
  • Suspend distribution of the windows-slaves-plugin
  • accounts.jenkins.io admin access for smerle
  • Upgrade to Kubernetes 1.29
  • [infra.ci.jenkins.io] Docker images CD failing

• Closed as not planned:

  • i lost my user name and password
  • [infra.ci.jenkins.io] 2024-08-31 Expiration of the Terraform backend credentials for the Terraform project azure

• Work in progress:

  • Ubuntu 24.04 campaign
    • Delayed until end of year (and curl bug fixed)
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • We need to (re)solve the update problem, while avoiding using an Azure CSI file PVC (SMB/CIFS locks)
    • Safer to use azcopy to copy data in the shared volume
    • Keep the “every 72h” frequency
    • Could be a Kubernetes Cron Pod, or an infra.ci job
    • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
    • To host stats.jenkins.io GSoC 2024 project in jenkins-infra
    • Add JDK21 agents (build)
      • WiP: Windows SSH agents. Removing default JDK from packer image templates.
    • Migration left over from publicK8s to arm64
      • mirrorbits runs with arm64 since 8 days. One less VM to pay for
      • Next candidate is LDAP. Need LDAP image and version updates.
      • Let’s delay of 2 weeks.
    • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
      • Let’s delay of 2 weeks as we need to finish the Org. scanning helm chart for infra.ci.
    • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
      • No more 503 errors: we added a fallback.
      • A lot of mirrorbits fixes (geoip, configuration tuning, arm64). Working well.
      • Public ACP removed from cluster: we have space for network peaks now!
      • Brownout:
        • Proposal 1 hour this Friday (08:00am UTC for the first try) => mailings, blog post, status, IMs.
        • Idea is to have a 2nd of 1 hour with US next week
        • Then 1 full day brownout
      • Performance improvement on Redis (see issue below)

• Triage/ToDo (next milestone) (infra-team-sync-2024-09-03 Milestone · GitHub)

  • [private.vpn.jenkins.io] 2024-09-06 (September 2024) VPN CRL expires
    • add to milestone
  • Ask OSSPlanet for a Jenkins mirror
    • add to milestone
  • [get.jenkins.io, mirrors.updates.jenkins.io] Optimize Redis used by mirrorbits instances (costs, security and performances)
    • add to milestone
  • Gradle plugin uses a proprietary dependency
    • add to milestone to follow, but no infra action needed. No emergency.
  • GSoC:
    • Replacing existing stats.jenkins.io code with https://github.com/jenkins-infra/stats.jenkins.io
    • add to milestone (define a migration plan)
    • Multiple requests for GSoC 2024 Plugin Modernizer Tool
      • add to milestone
    • Adding GSoC project to jenkins-infra
      • add to milestone
    • RPU GSoC project with terraform integration on infra’s backend
      • add to milestone
  • [privatek8s] Migrate AKS cluster to the sponsored subscription
    • Let’s delay until mid-Septmber when we’ll see a plan for credits (AWS/DO/Azure)
  • Upgrade to Kubernetes 1.30
    • Delay at least until October