Infrastructure Team Meeting - October 29, 2024

[dduportal (Damien Duportal)]

2024 10 29 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.482 went well
   • This Week: 2.483 started on time
2. Tomorrow: LTS 2.479.1
   • JDK11 EOL, Spring Security 6, Jetty 12, etc.

Upcoming Calendar

• Next Weekly: 2.484
• Next LTS: 2.479.1 - tomorrow - Mark is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • Azure AD Application password for Azure VM agents in `cert.ci.jenkins.io` expires on `2024-11-06T00:00:00Z` by jenkins-infra-updatecli[bot] · Pull Request #861 · jenkins-infra/azure · GitHub
• Next major event:
  • Jenkins Elections - voting start 1st of November
  • 2 weeks “Releases break” at the end of calendar year
    • Delays LTS 2.479.3 to early January 2025
  • Jenkins Contributor summit - 31 January 2025 in Brussels, Belgium
  • FOSDEM - 1 and 2 February - Brussels, Belgium

Cloud Budgets

• Azure CDF:
  • July: $4,571 (invoice)
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $3665 (forecast at 4.1k)
• Azure Sponsorship (Microsoft Credits) - Remaining: $35826 ($64174 consumed) until May 2025
  • July: $10k consumed
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $11.5k (Forecast at ~12k)
• DigitalOcean - Remaining $15,638 (~5k consumed) until 02 January 2025
  • July: $176.01 (invoice)
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $181 (Forecast at ~$200)
    • slight outbound increase (archives.jenkins.io usage)
• AWS:
  • CloudBees:
    • July: $6.5k
    • August: $6.3k
    • September: $6.1k
    • October: $5.8 (forecast at ~6.2k)
  • Sponsored account
    • October: $138 (forecast at ~$150)

Notes

• Done:

  • Failed to load https://updates.jenkins.io/download/plugins/
  • Block user racerpenguin for Jira issue spam
  • [OSUOSL] Migrate lists.jenkins-ci.org to a new (OSUOSL side) system
  • jenkins/jnlp-agent-jdk8 Docker image pushed 5 hours ago
  • [terraform-aws-sponsorship] updatecli manifest for AWS terraform provider
  • [terraform-aws-sponsorship] updatecli manifest for aws/vpc module
  • [Update Center - Azure] HTTP/404 responses instead of redirections with error AH01630: client denied by server configuration

• Closed as not planned:

  • Issue with showing build parameters

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • See post mortem
    • Let’s use Apache for directory listing!

  • ci.jenkins.io to AWS ([AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)):

    • Network: [ci.jenkins.io] Define virtual networking for AWS
      • Network ACL (subnet in and out) + SG (intra- subnets)
    • Controller: [ci.jenkins.io] Move controller (VM) to AWS
      • Sizing the machine: half CPU and memory, let’s use Graviton (arm64)
      • Simplifying (one network interface, one DNS entrypoint aws.ci.jenkins.io, only one disk)
      • Blocked by networking
    • VM Agents - [ci.jenkins.io] Move ephemeral VM agents to AWS
      • Linux AMI successfully validated by Jay using the ec2 plugin (but with default VPC and public IP so no config as code persistence for now)
      • Blocked by networking so mostly packer-images work:
        • [packer-images] track Linux base images with updatecli
      • (packer-images): Add Garbage Collector for AWS
        • Delayed (back to backlog) until Jay is back
    • Container Agents - [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • WiP on the initial Terraform resources to create EKS.
        • Studying the doc. to see what changed since 1 year ago.
        • Need to pair

  • GeoIP database updates ([get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart)

    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
      • We have our first Docker image in DockerHub with its own GitHub repository
      • Next step: the helm chart to setup the PoC CronJob

  • GSoC:

    • Delayed the terraform request to mid-November
    • Multiple requests for GSoC 2024 Plugin Modernizer Tool
      • Need a jenkinsci admin to transfer repository, next week to meet with Valentin timeline
    • Adding GSoC project to jenkins-infra
      • Need the user to transfer repository

  • Artifact Caching proxy issue (Pipeline can’t find the remote artifactory)

• New Issues (to triage):

  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • [terraform-aws-sponsorship] updatecli manifest for outbound ips
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • [ci.jenkins.io] Move ephemeral Linux containers to AWS
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)