Infrastructure Team Meeting - January 07, 2025

[dduportal (Damien Duportal)]

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• @hlemeur (Hervé Le Meur)
• @aayanmtn (Aayan Mateen)

Announcements

1. Jenkins Weekly Releases
   • 2024-12-17: 2.490 released with no hiccups
   • 2024-12-19: 2.491 (unplanned) released to address a regression in 2.490
   • This Week: 2.492 started on time earlier today, after the 2 weeks break

Upcoming Calendar

• Next Weekly: 2.493, 14 Jan. 2025

• Next LTS: 2.479.3, 8 Jan 2025 (tomorrow). Kris Stern is release lead.

• Next Security Release as per jenkinsci-advisories: N.A.

• Upcoming credentials expirations (~3 weeks):

  • 2025-01-12: => @smerle
    • Azure SP to deploy docs.jenkins.io - New end date for `docs.jenkins.io` File Share service principal writer on `infra.ci.jenkins.io` (current: 2025-01-12T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #909 · jenkins-infra/azure · GitHub
    • Azure SP to deploy javadoc.jenkins.io - Azure File Share Principal `javadoc.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-01-12T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #910 · jenkins-infra/azure · GitHub
    • Azure SP to deploy www.jenkins.io - Azure File Share Principal `www.jenkins.io` on `trusted.ci.jenkins.io` expires on `2025-01-12T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #911 · jenkins-infra/azure · GitHub
  • 2025-01-13: => @dduportal
    • Azure SP for ci.jenkins.io to spin up Azure VM Agents - Azure AD Application password for Azure VM agents in `ci.jenkins.io` expires on `2025-01-13T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #913 · jenkins-infra/azure · GitHub
  • 2025-02-02: repo.jenkins-ci.org (Artifactory) SSL certificates expires - SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025 · Issue #4477 · jenkins-infra/helpdesk · GitHub => @dduportal
    • If KK is not available, we can have a temporary Let’s Encrypt certificate

• Next major event:

  • Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025
    • Need to work on the Infra presentation and breakout session
      • accounts.jio
      • JDK21/25
      • etc.

Cloud Budgets

• Azure CDF:
  • October: $4.0k (invoice)
  • November: $4.3k (invoice)
  • December: $4.4k
  • 2024: $53.333 for the year (2023 was ~ $120.000)
  • January: $892 (forecast at $4.0k)
• Azure Sponsorship (Microsoft Credits) - Remaining: $30447 ($89553 consumed) until May 2025
  • October: $12.9k consumed
  • November: $13k
  • December: $9.5k
  • 2024: $87.000 for the year
  • January: $1.6k (forecast at $8k)
• DigitalOcean - Remaining $15,214.08 until January 02, 2026
  • October: $195.67 (invoice)
  • November: $146 (invoice)
  • December: $192 (invoice)
  • 2024: $5.000 for the year
  • January: $42 (forecast at $185)
• AWS:
  • CloudBees:
    • October: $6.4k
    • November: $3.9k
    • December: $540
    • 2024: $81.680 (2023: $136.627)
    • January: $105 (forecast at $581)
  • Sponsored account (~$58k credits lefts)
    • October: $178
    • November: $482
    • December: $595
    • 2024: irrelevant
    • January: $157 (forecast at $790)

Notes

• Done:

  • Support
    • AKS outages (publick8s, privatek8s, ci.jio-agents-1 and infra.ci.jio-agents-1) - OverconstrainedZonalAllocationRequest error when upgrading
    • Add ci.jenkins.io GitHub app to jenkins-usage-stats
    • Add mawinter69 to hosting team
    • To request Jan to be added to Jenkins core PR reviewers
    • CI/CD check stuck for a PR at stats.jenkins.io
      • Caused by expired credential (see below)
    • [JIRA] Change default assignee for plugin
  • Keep platform up to date
    • [Terraform] Expiration of credentials for Backend States and Cloudflare API tokens the 2024-12-26
    • Bump Terrafrom from 1.9.x to 1.10.x
    • Deprecate the Helm Chart docker-registry-secrets
    • [infra.ci.jenkins.io] Azure Service Principal Credential used to deployplugins.jenkins.io expires the 2025-01-08
    • [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy contributors.jenkins.io expires the 2024-12-18
    • [infra.ci.jenkins.io] Azure Service Principal Credential used to deploy stats.jenkins.io expires the 2024-12-18
    • [infra.ci.jenkins.io] Azure Service Principal Credential used to spawn Azure VM agents expires on 2024-12-18
    • Cleanup after accidents in Revoke accesses for Herve (temporary)
    • Implement artifact caching proxy for maven-hpi-plugin
  • ci.jenkins.io in AWS:
    • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents

• Closed as not planned:

  • Unable to Download Update-Center.json for Jenkins LTS 2.479.2
  • I need to create an account on jenkins for integrating with Jira
  • Cannot login to my account

• Work in Progress:

  • Support

    • Create build for jenkinsci/winp on release ci server
      • @dduportal
    • New repo scoop-bucket for app manifest distribution
      • Can we do it during FOSDEM all together? Let’s Ask Valentin
    • Foreign releases in public repository
      • @dduportal (as it requires Artifactory admin permissions)
    • Instability of artifact-caching-proxy
      • Remove from milestones until ci.jenkins.io to AWS (need to comment it)
    • build failure with useArtifactCachingProxy=true and dependency with version range
      • Remove from milestones until ci.jenkins.io to AWS
    • dnf5 update fails with gpgcheck=1
      • Stale (Damien to check with Tim and Mark)
      • Low priority
      • Should we work on this at FOSDEM?

  • Keep platform up to date

    • Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
      • Need to ping KK and Andrew for finishing 2024 stats
      • Thanks @timja for taking care of the Golang CLI part (the steps usually done by Andrew) which might help to solve

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • cluster created with ACP, datadog, etc.
      • @smerle is working on the ci.jenkins.io agents
      • Last task is to have ACP available through private LB (forthe VM agents)
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • Windows work, need to set it as code
      • Potential Optimization on Windows boot time
      • Need to provide all templates (same as current ci.jio)
    • [ci.jenkins.io] Move controller (VM) to AWS

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Decommission the old updates.jenkins.io Apache service. TODO.

• New Issues (to triage):

  • [terraform-aws-sponsorship] updatecli manifest for ami_release_version used by eks_managed_node_groups

    • => TODO @jay

  • [terraform-aws-sponsorship] updatecli manifest for cluster addon versions

    • => TODO @jay

  • [terraform-aws-sponsorship] updatecli manifest for isra_role module versions

    • => TODO @jay

  • SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025

    • => TODO @dduportal (as it requires Artifactory admin access)

  • Artifactory outdated maven-metadata.xml for public/com/github/jnr/jnr-posix/

    • => TODO @dduportal

  • AzureEdge.net CDN shuts down 15 Jan 2025

    • => TODO @smerle + @aayan

  • Issues with Gatsby deployment after dependency upgrades via PR for jenkins-infra/stories

    • => TODO @dduportal

  • [Update Center] HTTP/404 on /current/updates/*.json* links

    • Stale

  • External user struggling to submit story to stories.jenkins.io

    • Stale

  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org

    • Stale

  • [ci.jenkins.io] Set up an ECR pull through cache

    • TODO => @dduportal + @smerle

  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS

    • Stale

  • [ci.jenkins.io] Move ephemeral Linux containers to AWS

    • TODO => @smerle

  • Switch agent (java home) to JDK21 default

    • Stale

  • Switch default JDK to 21 for pipeline libraries

    • Stale

  • Switch default JDK to 21 for build tools

    • Stale

  • Move controllers to JDK21 (runtime)

    • Stale

  • Move agents to JDK21 (runtime)

    • Stale

  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

    • Stale (FOSDEM?)

• ToDo (next milestone) (infra-team-sync-2025-01-14 Milestone · GitHub)