Infrastructure Team Meeting - June 04, 2024

[dduportal (Damien Duportal)]

2024 06 04 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• Jay Reddy

Announcements

1. Weekly:
   • Release process started, still in progress (this meeting being 2 hours earlier)
2. New weekly meeting time today, let’s try it (12:00pm UTC)

Upcoming Calendar

• Next Weekly: 2.462, Tuesday 11 June 2024
• Next LTS (2.452.2):
  • RC last week
  • Wed. June 12th, 2024
• Next Security Release as per jenkinsci-advisories: None.
• Upcoming credentials expirations (~3 weeks):
  • No new expiration for the upcoming 3 weeks (current have issues in milestones, see below)
• Next major event:
  • Cd Mini summit, coordinated by Olivier Vernin, Vienna Austria - Sep. 2024
    • Bruno will be there!

Cloud Budgets

• Azure (CDF paid)
  • April: $4,550 (invoice)
  • May: $4,340 (4.240 reported + $100 monthly support)
  • June (current): $425 consumed (Forecast at ~3.8k)
• Azure Sponsorship (Microsoft Credits)
  • Global Status:
    • Credits left: $25,114 until 31 August 2024
  • April: $2,041
  • May: $5k consumed
  • June (current): $510 consumed (Forecast at ~4k)
• DigitalOcean
  • Global Status:
    • Credits left: $16,503 until 2nd January 2025
  • April: $840
  • May: $648 consumed
  • June (current): $17 consumed (Forecast at ~$250)
• AWS:
  • CloudBees:
    • April: $9,782
    • May: $8,281 consumed
    • June (current): $704 consumed (Forecast at ~$7.9k)
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • tester deverlor account access is denaild how to resolve
  • Remove user veronapressbuymagicmushrooms
  • Bump Maven to 3.9.7
  • Renew Docker Open Source Program sponsorship for 2024-2025
  • Packaging job on ci.jenkins.io never completes - retries fail
  • Request Temporal “Admin Read” Access for https://ci.jenkins.io/
  • nexus-jenkins-plugin bundles proprietary dependency

• Closed as not planned:

  • jenkin login issue
  • account recovery
  • ftp.halifax.rwth-aachen.de got blocked
  • Jenkins can’t start after April release
  • Salesforce authentication failed: Cannot run program “nohup” CreateProcess error=2, The system cannot find the file specified.

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • WIP on update_center2, testing the new “publish.sh” script (with feature flag and env. variable in credential out of the script)
      • Would open the possibility to have a staging environment in the future
  • Renew Azure credential for trusted.ci before expiration 8 of june
    • Done: automate future update with updatecli (and improve notification over “only calendar notifications”)
    • WiP: PR opened, need to perform the rotation. Planned tomorrow (Wed., morning EU), see message in #jenkins-infra => @smerle
    • As per @timja issue, we could envision using Azure Managed identity
  • Add a new private kubernetes cluster in the new sponsored azure subscription
    • WiP: drafting plan for the new cluster from what we recently did on ci.jenkins.io-agents-1 (private control plane, new CNI azure-overlay network, etc.)
      • First network, then Azure resources, then kubernetes management and finally Jenkins setup for agents
  • Enable 2FA on jenkinsci npm account
    • Password is currently on the jenkins-infra 1 Password
    • @lemeurherve want to set up the 2FA using our oathtool method
    • @smerle and @dduportal will test the new 2FA once done
  • Add .war.asc to get.jenkins.io
    • Status: Stalled
    • Let’s delay of 2 milestone (too much work in progress for Mark)
  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release
    • Done: Storage (disk, PV and PVC)
    • WiP:
      • disk access permission in Azure for AKS to read it
      • Migration rsync pod
    • Todo:
      • Plan operation once all is working as expected
      • Then release.ci and infra.ci as next candidates \o/
  • New Jenkins mirror in Romania by RCS&RDS
    • Planned for summer 2024 => let’s move this issue out of the milestones
  • New Jenkins mirror in Romania by Hostico
    • They asked for our outbound IP => gave them
    • Waiting for their (filtered) rsync server
  • packer-images: git version not always available simultaneously on apt ppa for arm64 and amd
    • Status: stalled as per @dduportal request to work on other tasks
    • WiP status: on the VM, git package is present (old version). Shall we override its content with our compiled git, or build and install our Deb package instead?
  • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts
    • Stalled => @dduportal

• ToDo (next milestone)

  • Change trusted.ci to use Managed Identity authentication for authenticating to Azure #4118 => legit, but backlog
  • [reports.jenkins.io][infrastructure data API] Add outbound IPs for get.jenkins.io #4114 => @dduportal
  • Expiration of the Digital Oceans PATs [10 June 2024] #4111 => @smerle , with @dduportal as fallback
  • Streamline Maven versions across the infrastructure #4110 => @dduportal