Infrastructure Team Meeting - January 21, 2025

[dduportal (Damien Duportal)]

2025 01 21 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last week: 2.493 released with no issues.
   • This Week: 2.494 started on time today
2. Azure Outages impacting Container Instances
   • Reminder:
     • [Incident] Windows build of plugins don't start on `ci.jenkins.io` · Issue #4490 · jenkins-infra/helpdesk · GitHub
     • Scope: only Windows container builds on ci.jenkins.io
     • Communication done last week after team meeting (including installation of customize-header plugin)
     • We are still in “degraded” mode: the temporary fix (using VM agents instead of containers) stays until migration to AWS => w
   • Issues to create:
     • Write down procedure to switch labels to VM (without breaking JDK selection) => @dduportal
     • Monitoring should alert us when agent allocations starts to fail => @dduportal
       • Increase build retention for jenkins-infra-plugin and the Infra/acceptance-tests
       • Datadog monitor (for both builds): status? build time?
       • See if pattern emerges to trigger alerts
   • New impact detected by @en3hD3iMRx6_6IXLNY0Rag on Jenkins Core Tests which times out around 6h. Short term: if no immediate solution disable test until we’ve migrated to AWS anbd then re-analysis.
3. No team meeting the 4 February 2025 (post-FOSDEM)

Upcoming Calendar

• Next Weekly: 2.495, Tuesday 28 January 2025
• Next LTS: 2.492.1 - (5 Feb 2025) - Jeremie Playout + Mark Waite as release leads
  • Damien will be available (train is Tuesday 4 Feb., not 5 as initially stated)
  • If we met requirements for Jeremie, we can start early
• Next Security Release as per jenkinsci-advisories:
  • Yes: tomorrow (22 January) - https://groups.google.com/g/jenkinsci-advisories/c/Vj6x6B3XCQo - plugins only
• Upcoming credentials expirations (~3 weeks):
  • repo.jenkins-ci.org TLS certificate: SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025 · Issue #4477 · jenkins-infra/helpdesk · GitHub
    • Damien and Mark still have to sync to see if we can do it without KK
    • LE as fallback
• Next major event:
  • Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025

Cloud Budgets

• Azure CDF:
  • October: $4,0k (invoice)
  • November: $4,3k (invoice)
  • December: $4,4k
  • January: $2,905 (forecast at $4.5k)
    • Increase in DNS billing in our zones
    • Increase on trusted.ci (not sure why: crawler failures?) agent
• Azure Sponsorship (Microsoft Credits) - Remaining: $24,714 until May 2025
  • October: $12,9k consumed
  • November: $13k
  • December: $9,5k
  • January: $7.4k (forecast at $10,8k)
• DigitalOcean - Remaining ~$15k until January 02, 2026
  • October: $195.67 (invoice)
  • November: $146 (invoice)
  • December: $192 (invoice)
  • January: $146 (forecast at $216)
• AWS:
  • CloudBees:
    • October: $6,4k
    • November: $3,9k
    • December: $540
    • January: $344 (forecast at $530)
  • Sponsored account (~$58k credits lefts)
    • October: $178
    • November: $482
    • December: $595
    • January: $578 (forecast at $890)

Notes

• Done:

  • Support
    • new Jira Spam account
    • [ci.jenkins.io] Install customizable-header to allow providing system message banner everywhere
  • Keep platform up to date
    • [cert.ci.jenkins.io] Service Principal used by cert.ci.jenkins.io to spawn Azure agents expires on 2025-02-02
    • [terraform-aws-sponsorship] updatecli manifest to track terraform module karpenter version
    • [terraform-aws-sponsorship] updatecli manifest to track terraform provider versions
    • [terraform-aws-sponsorship] updatecli manifest for cluster addon versions
    • [terraform-aws-sponsorship] updatecli manifest for irsa_role module versions
  • ci.jenkins.io to AWS
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)

• Closed as not planned:

  • The anti-spam system was triggered
  • Request for Assistance with Anti-Spam System Blocking Registration

• Work in Progress:

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Move controller (VM) to AWS
      • AWS CLI and kubeconfig installed
      • Next steps:
        • Fix certbot installation error
        • Set up TLS and hostname aws.ci.jenkins.io (instead of CNAME ci.jenkins.io) for tessting
    • [ci.jenkins.io] Move ephemeral Linux containers to AWS
      • JCasc config applied to both existing Linux Kubernetes Agents Clouds (default and BOM)
      • Karpenter installed and manually tested with success
      • Next steps:
        • Create Karpenter NodePools and test Pods are started
        • Fix controller HTTPS / DNS issues (see controller issue) and validate agents are starting
        • Finally, test a plugin build and a BOM build
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • Windows with SSH (as WinRM is too slow), both 2022 and 2019 works well.
      • Note: EC2 Fast Lauch (cost/speed)
      • ACP (in EKS) is reachable by the VM agents through an internal LB
      • Next steps:
        • Set up ACP JCasc and validate for both Linux and Windows
        • Apply the full JCasc config (with all agent templates)
        • Finaly test a few jobs requiring VMs: Docker Linux, Docker Windows, ATH, and a plugin using VMs explicitly (docker-workflow?)
    • Nexts steps:
      • Set up EKS Windows nodes instead of ACI ([ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS · Issue #4318 · jenkins-infra/helpdesk · GitHub)
      • ECR ([ci.jenkins.io] Set up an ECR pull through cache · Issue #4321 · jenkins-infra/helpdesk · GitHub)
      • JENKINS_HOME rsync initial copy Azure → AWS
      • Plan ETA for migration
  • Support
    • [Incident] Windows build of plugins don’t start on ci.jenkins.io
      • Stale until we migrated to AWS (see above)
    • Add Jeremie Playout as a Jenkins LTS release lead
      • Should only need to add a few users to the release-team GH’s group in jenkins-infra
    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
      • Stale: waiting for feedback from KK
  • Keep platform up to date
    • ci.jenkins.io: upgrade datadog plugin from 8.x to 9.x
      • Stale (waiting for AWS migration tests to be finished)
      • ETA this Thursday?
    • [terraform-aws-sponsorship] updatecli manifest for ami_release_version used by eks_managed_node_groups
      • WiP: @jay blocked by missing IAM permissions, @dduportal to unblock
    • SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025
      • Stale: Mark and Damien to meet
  • Artifactory
    • Stale (Damien alone), need to send email to developers to start discussion.
    • ETA: post fosdem
    • Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/
    • Foreign releases in public repository
  • Monitoring
    • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • Update Center:
    • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
      • Decommisiong staled

• New Issues (to triage):

  • Deploy jenkins-protoype on Netlify
    • Keep in Stale, need to answer with ETA after FOSDEM and discuss it in FOSDEM
  • [Update Center] HTTP/404 on /current/updates/*.json* links
    • Stale
  • External user struggling to submit story to stories.jenkins.io
    • Stale
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2025-01-28 Milestone · GitHub)

  • [trusted.ci.jenkins.io] Crawler fails to publish new tools metadata due to an S3 ↔ Cloudflare R2 error s3:PutObject NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER not implemented
    • Oracle Patch day: if we cannot find a quick solution, we’ll have to perform a one time manual update to ensure Oracle updates are pushed to the tools
  • Issues with Gatsby deployment after dependency upgrades via PR for jenkins-infra/stories
  • [terraform-aws-sponsorship] updatecli manifest to track karpenter helm_release version
  • [terraform-aws-sponsorship] updatecli manifest to track AWS Load Balancer helm_release version
  • jenkins.io SPF fail
  • Support [skip ci] on default branch
  • Downgrading a stats.jenkins.io maintainer to having no rights
  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS