Infrastructure Team Meeting - June 11, 2024

[dduportal (Damien Duportal)]

2024 06 11 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• Jay Reddy

Announcements

1. Weekly 2.462
   • Release process started, we’ll watch it
2. Meeting time: still good?
   • LGTM for everyone: we keep it like this
3. Azure deprecates Kubernetes 1.27 the 15th July 2024 (ref. Supported Kubernetes versions in Azure Kubernetes Service (AKS). - Azure Kubernetes Service | Microsoft Learn)
   • Issue to open => @dduportal (to include changes to clusters from last month) - Upgrade to Kubernetes 1.28 · Issue #4144 · jenkins-infra/helpdesk · GitHub
     • Scope for this milestone: issue, changelog review, proposed timeline and kubectl
4. Russian users
   • All russian ISPs have blocked connection to the Aachen University
   • New Romanian mirror opened yesterday that should be the default for Russian users
   • Looks like there is a Yandex mirror (inside Russian networks) => could be an additional solution. It’s worth checking with them to add these mirrors.
     • Any recollection? It doesn’t look like any of us worked on that as per Mark
     • Let’s contact them! => issue by @dduportal to track this work

Upcoming Calendar

• Next Weekly: 2.463, Tuesday June 18
  • Will require Java 17 or newer (see blog post)
  • Docker Packaging still uses JDK11 for the release build process (see docker-packaging/Dockerfile at 3fb7d59e10019e895cce1173e352f7600a91ac79 · jenkins-infra/docker-packaging · GitHub). We need to upgrade the JDK here.
    • Should we go with JDK17 or JDK21?
    • Need updatecli process to track this new one
    • @dduportal opens issue
    • Warning: how to ensure upcoming LTS lines between next week and 30 October 2024 (with JDK11 dropped) sticks to JDK11?
• Next LTS (2.452.2): tomorrow: Wed. June 12th
  • Kris Stern is release lead, Kevin Martens completed changelog
• Next LTS baseline selection: Wed June 26th
  • Will choose a weekly release that supports Java 11 (2.462 or earlier)
    • LTS 2.462.1 must continue Java 11 support
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • Service Principal used by infra.ci.jenkins.io to spawn Azure agents expires on 2024-06-30 => issue to create
  • Cloudflare API + R2 token are expiring July 2 => issue to create
• Next major event:
  • CD Mini summit, coordinated by Olivier Vernin, Vienna Austria - Sep. 2024
    • Bruno will be there!
    • Part of Open Source Summit Europe
  • DevOps World 2024
    • Register at the site to be informed of dates and details as available

Cloud Budgets

• Azure (CDF paid)
  • April: $4,550 (invoice)
  • May: $4,339 (invoice)
  • June (current): $1388 consumed (Forecast at ~4k)
• Azure Sponsorship (Microsoft Credits)
  • Global Status:
    • Credits left: $23,868 until 31 August 2024
  • April: $2,041
  • May: $5k consumed
  • June (current): $1747 consumed (Forecast at ~4.7k)
• DigitalOcean
  • Global Status:
    • Credits left: $16,300 until 2nd January 2025
  • April: $840
  • May: $648
  • June (current): $55 consumed (Forecast at ~$200)
• AWS:
  • CloudBees:
    • April: $9,782
    • May: $8,281
    • June (current): $2173 consumed (Forecast at ~$6.7k)
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • Block spammer cristaldibaroldy824@gmail.com
  • Missing contributor spotlight update for Vandit Singh
  • Timeout to ftp.halifax.rwth-aachen.de
  • [RPU] builds stuck in trusted.ci.jenkins.io
  • Expiration of the Digital Oceans PATs [10 June 2024]
  • Renew Azure credential for trusted.ci before expiration 8 of june
  • Enable 2FA on jenkinsci npm account
  • New Jenkins mirror in Romania by Hostico
    • We need to add them on jenkins.io’s website
  • Possible change to ci.jenkins.io’s linux agents causes core test failure
  • Replace Blue Ocean in default display URL (or remove the Blue Ocean plugins)
    • YAML of JCASC and Puppet 6.x do not use the same YAML parsing RFC

• Closed as not planned:

  • Can’t install or update plugins

• WiP:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • update-center2 PR is ready to review with tons of testing.
      • Need JenSec review to ask again, or if they delegate to us the process as it is only publication (not generation)
    • Todo:
      • Prepare performance test (handed over from Herve)
      • Prepare new update-center PRs to add azure bucket and new R2 bucket
      • Prepare work to have puppet managing VM for UC mirror
  • Use JDK21 Platform-wide
    • EPIC to track all tasks
    • See Add JDK21 agents (build) · Issue #4124 · jenkins-infra/helpdesk · GitHub (below) as first step
  • [reports.jenkins.io][infrastructure data API] Add outbound IPs for get.jenkins.io
    • Add Hostico manually
    • Add automated outbound from the new “Terraform” report export
  • Streamline Maven versions across the infrastructure
    • No work done, Should we prioritize the JDK upgrade instead for weekly 2.463?
      • Is Maven 3.8.x good enough?
      • If we have time yes, otherwise let’s delay
  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release
    • WiP on testing data migration
      • The “temp” pod is not scheduled on the same node as the controller on which to test backup/restore.
        • Need to work on affinities/selectors/tolerations
  • Add a new private kubernetes cluster in the new sponsored azure subscription
    • Cluster and node pools created on the sponsored subscription
    • WiP: adding cluster under kubernetes-management to prepare Jenkins management of agents

• ToDo (next milestone) (infra-team-sync-2024-06-18 Milestone · GitHub)

  • Add JDK21 agents (build) · Issue #4124 · jenkins-infra/helpdesk · GitHub
  • To host stats.jenkins.io GSoC 2024 project in jenkins-infra · Issue #4132 · jenkins-infra/helpdesk · GitHub => @lemeurherve
  • Cannot publish my plugin anymore · Issue #4133 · jenkins-infra/helpdesk · GitHub => @dduportal one last try and then escalate to Mark if no solution