Infrastructure Team Meeting - February 20, 2024

[dduportal (Damien Duportal)]

2024 02 20 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Next week: meeting cancelled, next team meeting will be 5 March 2024
2. Weekly 2.446
   • Release process successfull
   • Package successfull: Mirrorlist /windows/2.446/jenkins.msi
   • Docker Image:
     • 2.445 tag was recreated, triggered rebuild (git ref changed) and GH release republished
     • 2.446 first build failed, waiting for 2.445 to be finished (because of the “latest”) before retriggering
   • Changelog: to be finished later today
3. Ampere computing has lent 2 arm64 servers to the Jenkins project
   • Hosted at @en3hD3iMRx6_6IXLNY0Rag house
   • Server class machines with quite some resources
   • We have to wait early March and Mark’s return from holidays to get started
   • Thanks Ampere!

Upcoming Calendar

• Next Weekly: 2.447 next week (27 Feb. 2024)
• Next LTS: tomorrow (21 Feb. 2024) 2.440.1, Alex Brandes is release lead
  • Stéphane is (primary) lead on infra, Damien is backup
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • SCaleX (Los Angeles) in 17-19 March. Alyssa, Basil and Mark will be there

Notes

• Done:

  • Decouple incrementals-publishers from ci.jenkins.io for permissions check
  • Mirroring of https://packages.atlassian.com/mvn/maven-public/
  • update updatecli manifest on sharedtools to add a condition for golang with packer-images versions match
    • Closeable, to be verified by @lemeurherve

• Closed as not planned:

  • unbale to install the jenkins plugin name Role-based Authorization StrategyVersion 689.v731678c3e0eb_
  • Minimum Jenkins version required to support Mac ARM64 architecture
  • Commons-text API : earlier version plugin present on latest version link

• Work In Progress:

  • Check if we could replace blobxfer by azcopy

    • SAS short-lived token (using Azure Service Principal) works very well \o/
    • New pipeline library function to take care of this generation (and limit risks as it generates token in a shell system, not on controller)
      • Customizable (token TTL, permissions, etc.)
      • Opportunity to create a new plugin (Azure SAS token ?)
    • contributors.jenkins.io is using it and it work
    • Azcopy is installed (and version tracked) on the VMs (pkg.origin and agent.trusted)
    • WiP:
      • jenkins.io to use this system. But requires a storage account migration (v1 → v2, but also to Premium to reduce cost from ~70$ monthly to ~6-7 monthly). Might be the same for all other file shares.
      • Install the SAS token shell script in the VMs (pkg.origin and agent.trusted)

  • Upgrade to Kubernetes 1.27

    • Digital Ocean upgraded to 1.27
      • Removed terraform hacks
      • Upgrade is now managed as code
      • Note: we had to bump to latest 1.26 patch before the 1.27 upgrade (this patch update did not happen due to our cert-manager installation which had blockers as per DigitalOcean UI)
    • kubectl is now kept up to date on the agent.trusted (for the new Update Center implementation with Cloudflare)
    • Next step: upgrading AWS EKS. Delayed to March.

  • Update Jira LTS from 9.4.x to 9.12.x

    • No news

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Unblocked. Resumed work on using the SAS token generation for this project.
      • Script added (as freestyle job), to be discussed (updatecli? in agent.trusted)
      • Azure SP to be created/updated (currently in error, to be changed)
    • Next step:
      • Migrate to premium storage (transaction costs)
      • Re-test it once and then ask JenSec again for review

  • infra.ci.jenkins.io on arm64 (controller and agents)

    • WiP on replacing docker-builder (e.g. webbuilder)
      • PATH issue fixed (ruby 3.0 on arm64 polluting our systems)
      • Working on adding Ruby 3.2.x version (windows, updatecli, default version)
      • infra-reports should benefit from this change
    • Next step:
      • NodeJS/NPM and other consumers of “webbuilder” on infra.ci
    • Note: this will create a temporary discrepancy between ci.jenkins.io and infra.ci.jenkins.io. builds

  • Add a new private kubernetes cluster in the new sponsored azure subscription

    • Delaying to March

  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra

    • Let’s run the fix ASAP and then delay to March

  • [uplink] Download failing for JavaSystemProperties with error: missing chunk number 0 for toast value xx in pg_toast_xxx

    • Outage today due to database being reindexed
    • Fixed by setting replicas to only 1 to avoid concurent database operations
    • We have to add a startupprobe to the helmchart (as it takes time to run)
    • Still working on cleaning up corrupted records. But not top priority as Daniel does not need this data anymore.

  • Unexpected delays building small plugin on linux agent

    • Kubernetes 1.27 is ok on DO. Worth re-enabling DO agents on ci.jenkins.io to check if it is still slow.
    • Delayed to March by default

  • Export download mirrors list to a textual representation

    • Fix is needed (to find another page than updates.jenkins.io “latest” redirection. get.jenkins.io should be used). get.jenkins.io provides a page with the list of all configured mirrors, but we have to find it.

=> Delaying the others to March

• removing the central cache has seemingly broken dependabot (sometimes!)

• [Jenkins Agents] Clean up deprecated JNLP arguments

• Revoke an OpenVPN cert for NotMyFault

• To host versioned jenkins.io docs on docs.jenkins.io

• Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io

• Migration left over from publicK8s to arm64

• Past Release sites are taking long time to load

• ToDo (next milestone) (https://github.com/jenkins-infra/helpdesk/milestone/[ID+1])

  • ftp.halifax.rwth-aachen.de blocked my provider => dduportal