Infrastructure Team Meeting - July 30, 2024

[dduportal (Damien Duportal)]

2024 07 30 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @jayfranco999 (Jay Reddy)

Announcements

1. Weekly 2.469 (23 July 2024) went well (no issues) and was on time
2. Weekly 2.470 (30 July 2024) started on time
3. Next meeting cancelled (6 of August 2024)
   • Next meeting: 13 August 2024

Upcoming Calendar

• Next Weeklies:
  • 2.471 (6 August 2024)
  • 2.472 (13 August 2024)
    • Will have upgrade of Jetty from 10 to 12 EE 8 => need to monitor and test heavily
• Next LTS: 2.462.1 (7 August 2024) Mark Waite leads
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • RPU Artifactory Admin token expires 13 Aug. 2024: [trusted.ci.jenkins.io] RPU Artifactory API token expires the `2024-08-13` · Issue #4203 · jenkins-infra/helpdesk · GitHub
• Next major event:
  • CD Mini Summit, September 19 in Vienna, Bruno Verachten talk accepted and Olivier Vernin will be there (con. leader)

Cloud Budgets

• Azure (CDF paid)
  • April: $4,550 (invoice)
  • May: $4,339 (invoice)
  • June: $4,287 (invoice)
  • July (current): $4218 consumed ( Forecast at ~4.5k)
    • Need to carefully check why did it increase
    • Issue to create to migrate privatek8s (to decrease bill)
    • Issue to create to migrate cert.ci and trusted.ci VMs (3) - (to decrease bill)
• Azure Sponsorship (Microsoft Credits) - Remaining: $68641 until May 2025 (instead of August 2024)
  • April: $2k
  • May: $5k consumed
  • June: $7.3k consumed
  • July (current): $9613 consumed ( Forecast at ~10k)
    • Issue to create about using spot instances to decrease costs (at least ask for quotas to Azure Support)
• DigitalOcean
  • April: $840
  • May: $648
  • June: $165.32
  • July (current): $160 consumed (Forecast at $165)
    • Issue to create for a “reference mirror”
• AWS:
  • CloudBees:
    • April: $9,782
    • May: $8,281
    • June: $5,862
    • July (current): $6k consumed (forecast 6.5k)
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • Delete spambot ‘lumiereboulevardquan9’
  • Delete users ‘samuel898’ and ‘krina234pestro’ and their comments
  • Delete Account
  • The ssh-credentials plugin released an hour ago is not visible from updates.jenkins.io
    • Cause by SSH restrictions below. Fixed.
  • Restrict SSH to all VMs from VPN
  • Install most recent JUnit plugin on ci.jenkins.io and friends
    • NB. might want to update it again due to minor fixups
  • stop using Tim Jacomb personnal account on maxmind for geoip
  • Dockerhub rate limit broke the www.jenkins.io CI build
    • New issue about an “ACP-like” docker cache for ci.jenkins.io
      • Decrease the outbound bandwidth from DockerHub, respecting their free service (“Tragedy of the commons”)
    • (Update after meeting) reopened and tracking the DockerHub mirror pull trough cache in it.
  • ci.j.io plugin jobs don’t trigger on branch scan
  • [cert.ci.jenkins.io] Jobs failing with TransportException: github.com:443 failed to respond when using git pipeline step with JGit tool
  • new.stats.jenkins.io slow to load
  • Adding an new mirror in Taiwan
  • New mirror in Japan

• Closed as not planned:

  • Downloads of git-changelist-maven-extension from plugin build failing

• Closed as not planned:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Mirror system is ready and balance requests geographically as expected
    • Functional tests started
      • jenkins-plugin-cli works perfectly
      • Hiccups from Jenkins Plugin UC UI but fixed
      • Last one: send everything to auzre.updates instead of updates in testing
      • Let’s start using this service for our controllers
    • Stress test to start
      • from a full day access.log of prod, but in 1 hour
      • Random stress test
    • Operational tasks
      • Ensure we collect metrics and logs for this application in Datadog
  • Temurin JDK upgrade July 2024
    • Waiting for LTS
    • And s390x JDK upgrades (pairing with Jay)
  • Add JDK21 agents (build)
    • ci.jenkins.io is now providing (Linux) maven17 and maven21 for VMs (with Docker)
      • Also arm64 VMs (but it does not work)
    • Next step: SSH agents maven17 and maven21 (Linux) for trusted.ci and cert.ci
    • And then: Windows
  • [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
    • WiP on job config, to be verified
    • otherwise we’ll split credentials
  • Remove 999999-SNAPSHOT version of Remoting from Artifactory
    • On hold, see Switch to renovate from dependabot for remaining dependencies by timja · Pull Request #9459 · jenkinsci/jenkins · GitHub
    • Basil asked to remove the faulty artifact to unblock
  • Ask OSSPlanet for a Jenkins mirror
    • On hold: No news since 1 week, let’s wait until end of month
  • [Plugin Health Score] Scores not computed - Getting logs from plugin-health.jenkins.io
    • On hold: Adrien is overwhelmed
  • Migration left over from publicK8s to arm64
    • Next steps are LDAP (need to run it locally) and mirrorbits (custom binary build)
  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
    • Did not work on this, resuming this week

• ToDo (next milestone) (infra-team-sync-2024-08-13 Milestone · GitHub)

  • VPN access to usage.jenkins.io for usage stats generation
    • Andrew need VPN access in order to reach usage.jenkins.io with SSH
  • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2024-08-13

• Mark Waite has a request:

  • James Brown works for a company doing machine learning on datadog traces
  • They propose free use of their services to OSS projects
  • 5 areas:
    • Pipeline observability
    • Outages, Incidents, etc.
  • Would require creating an API key for them, and their results can be private