Infrastructure Team Meeting - April 23, 2024

[dduportal (Damien Duportal)]

2024 04 23 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly release 2.455 is out (almost)
   • WAR and packages are
   • Changelog is out
   • Docker image is WiP (tag created, build/publish in progress)
2. Team days off
   • Limited availability:
     • Stéphane out this week
     • Hervé out tomorrow
     • Damien out Thursday and Monday
3. AWS sponsorship renewal for 2025
   • Schedule accelerated: deadline tomorrow
   • Mark will submit (check with board)

Upcoming Calendar

• Next Weekly: 2.456 (30 April 2024)
• Next LTS: 2.452.1 - (15 May 2024) - Alex Brandes is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • Mark speaking at MileHigh Agile in Denver (Colorado) in 2 weeks

Notes

• Done:

  • User vanellope blocked for junk issue report
  • Block spam generating user reloaddigital88@gmail.com
  • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.440.3
  • [infra.ci.jenkins.io] Terraform Azure job fails with Total node count 56 requires 71680 ports but only 64000 ports are available given 1 outbound public IPs
  • [infra.ci.jenkins.io] Azure AD Password expired for terraform-production (jenkins-infra/azure)
  • Let’s Encrypt Azure Service Principal passwords expired for cert.ci.jenkins.io and trusted.ci.jenkins.io
  • Migrate release.ci.jenkins.io to arm64 (controller only)
  • infra.ci.jenkins.io on arm64 (controller and agents)

• Closed as not planned:

  • MISMATCH
  • not able to login

• Work In Progress:

  • renew Service Principal Password before expiration for application sponsorship.ci.jenkins.io-tf-managed

    • Todo (later): improve updatecli PR to give directions (runbook) on what to do when merged
    • Todo (later): With this updatecli automated process: create an issue (with or without the PR) on helpdesk so we can track it for meetings (and replace calendar events)
    • Todo (later): factorize some of the code to reuse the same on all terraform repositories
    • Next step for this issue:
      • Approve, Merge and take care of the new credentials for ci.jenkins.io => @dduportal
      • New PR to add the “runbook” => dduportal

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • The “new” UC (azure.updates.jenkins.io) is missing some UC redirects (redirect by Apache today when requesting a dynamic UC with a query string)
      • Routing at reverse-proxy using query string is dangerous and slow => let’s avoid doing it on the (shared ingress
      • Besides, this “routing/redirect” logic is already available in the generated Apache2’s .htaccess => let’s not reinvent the wheel
      • Proposal: send everything to (the new) Apache2 and add a vhost fallback to a new DNS (mirrors.updates.jenkins.io) which sends remaining requests without query string to mirrorbits
    • Next steps:
      • Fix the above behavior to handle dynamic URLs (with a QS)
      • Then back to stress tests and JEP review
        • We have reports of hit rate and bandwidth from today’s logs

  • [AWS] Migrate ci.jenkins.io EKS clusters from CloudBees AWS account to Jenkins AWS (sponsored) account

    • IAM permissions pattern is going on the right direction
      • Stéphane and Damien have accounts with the expected “non root + MFA + assumeRole”
    • WiP:
      • Setting up technical Terraform IAM users
      • Start a first EKS cluster
        • IAM profile to be managed NOT by the terraform module
        • Name of the cluster is ci.jenkins.io-agents (or cijenkinsio-agents or cijio-agents)
        • Only 1 cluster instead of 2: ACP will be internal only on AWS

  • JDK patch upgrade campaign (April 2024)

    • Issue written to have a “map” of where to upgrade JDKs
    • Tools JDK should be up to date
    • WiP on controller/agents
      • Windows containers are next
    • Agents: let’s verify with @smerle 's list

  • [infra.ci.jenkins.io] Build of the jenkins-infra/docker-inbound-agents are failing since 20 March 2024

    • Fix test phase to allow JDK upgrades
      • Disable tests?
      • Rollback to cst 1.6.1 (known to work on Windows)?
      • Move to goss?

  • Release of jenkins/* container images on DockerHub errors with HTTP/429

    • Closeable for Jenkins Infra => move the last image changes to SIG platform

  • Support for Visual Studio in windows container image

    • Only Windows VM images (ci.j ephemeral agents)
    • Requirement for building winp on ci.jenkins.io
    • WiP:
      • PR from Alex Earl to install it in our agent templates. Windows 2019 works, WiP on Windows 2022 support (installation stuck on Win 2022), fix currently tested
      • Then release and deploy a new version to provide something to Pierrick
    • If Alex needs (and want) maintainer access to the repo + VPN access: let’s roll (no objection during team meeting)

  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release

    • Goal: save Azure (CDF paid!) money by using cheaper SSDs for jenkins homes
    • Starting with weekly.ci (process in the issue) to validate the process, before proceeding on infra.ci and release.ci
    • Done: Storage class created on both clusters
    • Todo:
      • weekly.ci (next week monday/tuesday)

  • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts

    • Damien need to send an email to request help from developers (still)

  • Move the jenkins-stats repositories to the jenkins-infra organization

    • Back to backlog (no team capacity)

  • migrate plugin-site-api generate-data from ci.jenkins.io to infra.ci.jenkins.io

    • Back to backlog (no team capacity)

  • New Jenkins mirror in Romania by RCS&RDS

    • Waiting for mirror provider feedback (HTTPS missinbg on their side)

  • New Jenkins mirror in Romania by Hostico

    • Damien takes it over to verify the FTP credentials are supported by mirrorbits

• Triage:

  • Design an automated update mechanism for the private repository jenkins-infra/terraform-states · Issue #4051 · jenkins-infra/helpdesk · GitHub => backlog
  • Add `.war.asc` to get.jenkins.io · Issue #4055 · jenkins-infra/helpdesk · GitHub => postponed to May
  • Plugin health JSON sometimes missing · Issue #4056 · jenkins-infra/helpdesk · GitHub => adding to current milestone
  • Packer images failed to build due to `(429) Too Many Requests` errors from Chocolatey registry · Issue #4059 · jenkins-infra/helpdesk · GitHub => postponed to May

• ToDo (next milestone) (infra-team-sync-2024-04-30 Milestone · GitHub)