Infrastructure Team Meeting - August 20, 2024

[dduportal (Damien Duportal)]

2024 08 20 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @kmartens27 (Kevin Martens)
• @jayfranc999 (Jay Reddy)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly
   • 2.472 released successfully last week
   • 2.473 started on time today

Upcoming Calendar

• Next Weekly: 2024-08-27 - 2.474 weekly release
  • Note: 2024-09-03 → weekly will have Jetty 12 EE9 (and Spring Security 6.x).
• Next LTS: 2024-09-04 - 2.462.2 - Alex Brandes is release lead
  • Next baseline selection - 2024-09-18
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2024-08-31: Terraform azure project backend credentials: [infra.ci.jenkins.io] `2024-08-31` Expiration of the Terraform backend credentials for the Terraform project `azure` · Issue #4251 · jenkins-infra/helpdesk · GitHub
  • 2024-08-31: release.ci.jenkins.io Azure Client to access secret Vault expires: [release.ci.jenkins.io] `2024-08-31` Azure Credentials for Core Release (Vault access) expires · Issue #4252 · jenkins-infra/helpdesk · GitHub
  • 2024-09-02: trusted.ci.jenkins.io Azure VM Client ID: [trusted.ci.jenkins.io] `2024-09-02` Credential for spawning Azure VM agents expires · Issue #4253 · jenkins-infra/helpdesk · GitHub
  • 2024-09-05: DigitalOCean PATs expires: [infra.ci.jenkins.io] `2024-09-05` Expiration of the Digital Oceans PATs used by the Terraform project `digitalocean` · Issue #4254 · jenkins-infra/helpdesk · GitHub
• Next major event:
  • DevOps World Virtual online September 17, 2024
    • Jenkins officers and board presentation and Q&A
  • CD Mini Summit in Vienna, September 19, 2024
    • Thanks to Olivier Vernin for leading
    • Thanks to Bruno Verachten for presenting

Cloud Budgets

• Azure (CDF paid)
  • May: $4,339 (invoice)
  • June: $4,287 (invoice)
  • July: $4,571 (invoice)
  • August: $2.9k consumed (Forecast at ~4.5k)
    • Still need to carefully check why did it increase
    • Merge publick8s x86 node pools → Migration left over from publicK8s to arm64 · Issue #3837 · jenkins-infra/helpdesk · GitHub
    • migrate privatek8s to sponsored account → [privatek8s] Migrate AKS cluster to the sponsored subscription · Issue #4250 · jenkins-infra/helpdesk · GitHub
    • Issue to create to migrate cert.ci and trusted.ci VMs (3) - (to decrease bill)
    • Issue to create to Use only 1 Redis managed service for both get.jenkins.io and updates.jenkins.io
• Azure Sponsorship (Microsoft Credits) - Remaining: $61328 until May 2025 (instead of August 2024)
  • May: $5k consumed
  • June: $7.3k consumed
  • July: $10k consumed
  • August: $6.6 consumed (Forecast at 10k)
• DigitalOcean - Remaining 16k$ until 02 January 2025
  • May: $648
  • June: $165.32
  • July: $176 consumed
  • August $131,23 (Forecast at $200)
    • Bandwidth increase
• AWS:
  • CloudBees:
    • May: $8,281
    • June: $5,862
    • July: $6.5k
    • August: $4k, forecast at $6.5k
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • Delete link https://issues.jenkins.io/browse/JENKINS-64860
  • Spam accounts:
    • Vandalism in Jira
      • Rate decreased: we can continue with the current method. poroblem from last weeks have been fixed
    • Block use nestorg780 due to issue spam on issues.jenkins.io
    • Delete spambot ‘watsicagaylord’
    • Cannot signup in jenkins
    • I’ve been blocked
  • Bad gateway message fails some ci.jenkins.io builds
  • “Connection reset” errors with artifact-caching-proxy.privatelink.azurecr.io
    • It was a mistaken hostname
    • Ego metric: ACP was close to 2 GBits/s for BOM build artifacts downloads.
      • Way better these Gbits/s out from our system than from Artifactory (reduced load)
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart

• Closed as not planned:

  • Missing dynamic update center for 2.452.4
  • NA

• Work in Progress (Milestone 123):

  • Add JDK21 agents (build)

    • JDK17 and JDK21 templates created and deployed for Linux Inbound, Linux SSH and Windows inbound (ci.jio, trusted.ci.jio, cert.ci.jio)
    • WiP for JDK17 and JDK21 templates on infra.ci (SSH launcher) => @jayfranco999
      • PR opened, fixed, ready to deploy
    • WiP Windows SSH templates => @dduportal
      • Adding environment variable works
      • But not changing Path (due to Windows OpenSSH behavior)
        • User/Machine env. level + restart SSH service
        • Fallback: remove default JDK from Windows VM templates

  • [infra.ci.jenkins.io] Azure VM agent init script not in sync with Puppet code

    • Found while working on Add JDK21 agents (build)
    • Datadog often fails to start during agent init phase
    • PR opened, ready to deploy as well

  • Suspend distribution of the windows-slaves-plugin

    • This plugin uses an outdated technique to launch agents. It should not work anymore except if you purchased extended MS support for Windows 2012…
    • Concern: will we break users setup if any has this (exotic) setup?
    • Discussion still in progress (ref. PR on update-center2)

  • plugin stats for bouncycastle-api are missing data.

    • Need coding “help” to solve it => not really a direct action can be done by the infra team
    • Removing from milestone, and even move it to the repository

  • On hold (no work done or waiting from someone outside):

    • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
    • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
    • [Plugin Health Score] Scores not computed - Getting logs from plugin-health.jenkins.io
      • Let’s remove from milestone: no actionnable for us

  • Migration left over from publicK8s to arm64

    • We are able to run mirrorbits on arm64 \o/
      • 2 method but we have to choose:
        • We build the binary ourselves
        • Debian package
      • Let’s move all mirorbits to arm64!
    • Last leftover: LDAP and we can get rid of x86 nodes

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Fixed the frequent 503 errors
      • Serve stale data (5 min old) instead of 503, thanks to fallback
    • Improved monitoring: collecting all kind of logs
    • Improved security (RO filesystem, no capabilities, etc.)
    • No errors using it with ALL of our controllers
      • Neither from MarkWaite’s network
    • Next step:
      • Use mirrorbits gRPC client instead of kubectl exec on trusted.ci
      • Add datadog monitoring on the expected URL to validate redirections and content
      • Generate missing index listing

• ToDo (next milestone) (infra-team-sync-2024-08-27 Milestone · GitHub)

  • Bump Maven to 3.9.9
    • Let’s move to ATH as “smoke test”
    • Campaign to proceed
  • Upgrade to Kubernetes 1.29
    • Let’s get started with kubectl
    • infra.ci-agents cluster to migrate end of week
  • To host stats.jenkins.io GSoC 2024 project in jenkins-infra
    • Need helm-chart update to implement the fix => @lemeurherve takes care of it
    • Almost end of GScoC: this is a nice last item!
  • Cannot release a new plugin with CD even if tokens are created and automatic PR on RPU merged
  • RPU GSoC project with terraform integration on infra’s backend
    • Need clarification: no milestone yet
  • [accounts.jenkins.io] Decrease Google indexing
  • Ubuntu 24.04 campaign
    • Still need to analyse the broken build for docker-packaging
  • [ci.jenkins.io] Bump datadog plugin from 7.0.1 to 7.1.2