Infrastructure Team Meeting - May 28, 2024

[dduportal (Damien Duportal)]

2024 05 28 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• Jay Reddy

Announcements

1. Weekly: 2.460
   • Changelog: merged!
   • WAR Released
   • Package is OK (stuck at mirror sync, but done)
   • Docker image: todo later today
2. Weekly new meeting schedule
   • Poll: Sondage - Jenkins Infra SIG weekly meeting time - Framadate
   • Let’s try 2 hours earlier than doay for the upcoming weeks
   • Don’t hesitate to raise your voice for other option such as alternating 2 different times to cover the whole globe
3. (minor) security advisory last Friday: https://groups.google.com/g/jenkinsci-advisories/c/ZAwn9ooELD0

Upcoming Calendar

• Next Weekly: 2.461, Tuesday 4 June 2024
• Next LTS (2.452.2):
  • RC this week
  • Final June 12th, 2024
• Next Security Release as per jenkinsci-advisories: None
• Upcoming credentials expirations (~3 weeks):
  • Azure credential for trusted.ci (8 June 2024)
    • @smerle creates the issue and add it to new milestone
  • Digital Ocean PATs (10 June 2024)
    • @dduportal creates the issue and add it to new milestone

Cloud Budgets

• Azure (CDF paid)
  • March: $4,398 (invoice)
  • April: $4,550 (invoice)
  • May (current): $3771 consumed (Forecast at ~4.3k)
• Azure Sponsorship (Microsoft Credits)
  • Global Status:
    • Credits left: $26,349 until 31 August 2024
  • March: $2,375
  • April: $2,041
  • May (current): $4,266 consumed (Forecast at ~4.7k)
• DigitalOcean
  • Global Status:
    • Credits left: $16,522.71 until 2nd January 2025
  • March: $938
  • April: $840
  • May (current): $645 consumed (Forecast at ~$714)
• AWS:
  • CloudBees:
    • March: $9,567
    • April: $9,782
    • May (current): $7706 consumed (Forecast at ~$8.8k)
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • ftp.halifax.rwth-aachen.de blocked
  • Packer-Images: bug within updatecli to update goss since exclusion of windows 2019 for vsstudio
    • YAML syntax error making updatecli fail (Go template inside the yaml, used by Goss) preventing us to create dependency bump PRs since 4 weeks
    • Updatecli fixed by changing the YAML parser (short term fix). Allowed us to release new image versions
    • Goss syntax fixes to make the YAML valid: opportunity to add a “failfast” and correct error code: now if goss fail on Windows, then the pipeline fail \o/
  • [ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account
    • Only cleanup was left: done!
    • We see the decrease on AWS (CloudBees) and DO bills while consuming in Azure (sponsored account)

• Closed as not planned:

  • sparshev account is unavailable in issues.jenkins.io
  • Blocked by mirror, unable to update plugins.

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Handover from @lemeurherve to @smerle done (thanks!)
    • Done:
      • Designed the next steps around production architecture:
        • Cloudflare R2: we already have a West-Europe bucket, we need a second one in US-East (same as PKG vm) to provide better latency for US users
        • OSUOSL VMs: if Cloudflare fails (whatever reason) we need a fallback. We have 2 unused VMs, with enough resources (CPU/Memory/disk) to be used for this use case
        • In case of terrible problem, we also can spin up a VM in DigitalOcean for fallback/alternative
      • Credentials for Azure HTTPD (new) - bucket tested with success!
    • Wip:
      • update-center2: we expect 4 PRs (1 already in draft)
        • Abstract away the environment (bucket URLs/credentials) from the publish.sh script to allow real life testing without impacting production and manage multiple sync targets per "kind " (e.g. rsync/azcopy/s3/R2) => @dduportal
        • Adding the azcopy httpd sync support (Draft already there) => everyone
        • New R2 bucket to be added => @smerle
        • Later: OSUOSL VMs (rsync)

  • New Jenkins mirror in Romania by RCS&RDS

    • They still need to add HTTPS on their server (and give us credentials for sync) => pinged them, still waiting

  • New Jenkins mirror in Romania by Hostico

    • Done:
      • Verified that we can specify user and password in the FTP or RSYNC inline URL (ftp://user:password@hostname/path for instance)
    • WiP:
      • Waiting for them to re-enable their FTP or rsync => @dduportal once they answer

  • Packaging job on ci.jenkins.io never completes - retries fail

    • Root cause: cluster change to Azure on ci.jenkins.io
    • Wip:
      • PR on the project to unblock the pod allocation
      • Choice: merge pod template definition between admin-defined and developer defined for this particular one (packaging Docker image in,stead of all in one) => @dduportal with @smerle

  • ftp.halifax.rwth-aachen.de got blocked

    • Closeable as C-Otto responded

  • packer-images: git version not always available simultaneously on apt ppa for arm64 and amd

    • Proposal: we are experimenting compiling our own git instead of relying on packages (wether ubuntu or PPA) which is safer.
    • WiP:
      • Compilation work very well and is really fast
      • Problem: on VMs, the meta-package ubuntu-server has git as a direct dependency
        • Solution 1: let’s override the git package even if present. Need exhaustive file override and might lead the package tree broken.
        • Solution 2: build our own deb package as a post-step of compilation
      • Cleanup of unused build dependencies
      • Updatecli manifest

  • Request Temporal “Admin Read” Access for https://ci.jenkins.io/

    • Carlos want to run a “Jenkins Cloud-Friendly” talks in June. He wanted to show different CNCF/Cloud-Native projects linked to a real life Jenkins instance.
    • Damien met Carlos, an we’re going to share links to the “public” setup we have for a few elements:
      • Azure VM and Azure Container based agents
      • Kubernetes agents
      • Former EC2 setup
      • Datadog setup
    • If needed we’ll meet to record Damien’s screen to give him more material
    • Note: with their Jenkins account, he can show builds logs with Pod template agent definition

  • Add .war.asc to get.jenkins.io

    • WiP by Mark and Basil

  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release

    • We changed Kubernetes volume from “Dynamic Provisioning” to “Static Provisioning” to control the disk (in Azure API) and its attributes such as type
    • We had to create a “dummy” storage class to ensure PV and PVC statically provisionned have the same “storage class” value (not empty and not default)
    • Next steps: plan operation with the rsync migration (preparation involved) => @smerle

  • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts

    • Nothing done (on Damien to send email)

• New issues:

  • Maven 3.9.7 => need an issue @dduportal
  • Start discussion about Jenkins core release with JDK17 and Maven 3.9.x => issue (@dduportal and @markewaite)
  • NPM Jenkins account - MFA => Enable 2FA on jenkinsci npm account · Issue #4073 · jenkins-infra/helpdesk · GitHub => @lemeurherve
  • Add a new private kubernetes cluster in the new sponsored azure subscription · Issue #3923 · jenkins-infra/helpdesk · GitHub => @smerle to see (least priority)
  • Replace Blue Ocean in default display URL (or remove the Blue Ocean plugins) · Issue #2833 · jenkins-infra/helpdesk · GitHub => @dduportal with Jay as secondary

• ToDo (next milestone) (infra-team-sync-2024-06-04 Milestone · GitHub)