Infrastructure Team Meeting - November 19, 2024

[dduportal (Damien Duportal)]

2024 11 19 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @kmartens27 (Kevin Martens)
• @MarkEWaite (Mark Waite)

Announcements

1. Jenkins Weekly Releases

   • Last Week: 2.485 went well
   • This Week: 2.486 (started on time at 10:30 UTC?)

2. We had a security advisory (plugins only) last week: https://groups.google.com/g/jenkinsci-advisories/c/UmiH1ZpJez4

Upcoming Calendar

• Next Weekly: 2.487 (26 Nov. 2024)
• Next LTS: 2.479.2 (27 Nov. 2024) - Kris Stern is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • Packer credentials in infra.ci (09 Dec. 2024) - Extend Azure AD Application password validity for packer-resources (current end date: 2024-12-09T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #874 · jenkins-infra/azure · GitHub
• Next major event:
  • Jenkins Elections - voting started 1st of November, registration reopened, over 50 votes already cast from 78 registered voters
  • 2 weeks “Releases break” at the end of calendar year
    • Delays LTS 2.479.3 to 8 Jan 2025
  • Jenkins Contributor summit - 31 January 2025 in Brussels, Belgium
  • FOSDEM - 1 and 2 February - Brussels, Belgium

Cloud Budgets

• Azure CDF:
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $4.0k (invoice)
  • November: $2.6k (forecast at $4.4k)
• Azure Sponsorship (Microsoft Credits) - Remaining: $27690 ($72310 consumed) until May 2025
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $12.9k consumed
  • November: $6.3k (Forecast at $9.9k)
• DigitalOcean - Remaining $15,519 ($4k consumed) until 02 January 2025
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $195.67 (invoice)
  • November: $105 (Forecast at $165)
    • Requesting extension in Dec.
• AWS:
  • CloudBees:
    • August: $6.3k
    • September: $6.1k
    • October: $6.4k
    • November: $3.7k (forecast at $6.1k)
  • Sponsored account
    • October: $178
    • November: $247 (forecast at ~$400)

Notes

• Done:

  • [release.ci.jenkins.io] 2024-11-20 Azure Credentials for Core Release (Vault access) expires
  • [trusted.ci.jenkins.io] 2024-11-20 Credential for spawning Azure VM agents expires
  • [infra.ci.jenkins.io] 2024-11-21 Expiration of the Digital Oceans PATs used by Terraform
  • Delete accounts.jenkins.io account
  • Block user michelle653burkes due to Jira issue spam
  • Block user purnima05 due to Jira issue spam
  • Block user pootler due to Jira issue spam
  • Block happywrencher due to Jira issue spam
  • ci.jenkins.io: upgrade datadog plugin from 7.x to 8.x
  • [terraform-aws-sponsorship] updatecli manifest for outbound ips
  • Move GSoC 2024 Plugin Modernizer Tool to jenkins-infra
  • RPU GSoC project with terraform integration on infra’s backend

• Closed as not planned:

  • Latest Workflow-CPS commit is not generating a new release

• Work in Progress:

  • New UC:

    • Base container jenkins servers - Plugins 404 errors
    • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
      • Nginx was logging rewrites: disabled for better performances
      • Shared file system tends to be slow suddendly for httpd containers => need to update
      • Globally a success: intermittent issues to fix but we are getting there

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • (packer-images): Add Garbage Collector for AWS
      • Distinct job (and faster build due to this change)
      • Works for AMI and snapshots
      • WiP on instances/Security groups
    • [ci.jenkins.io] Define virtual networking for AWS
      • EKS is on hold due to Stephane in sick days
      • Controller / subnets => VM for controller is on public subnet for IPv6
        • Network security is applied for controller and future VM agents
    • [ci.jenkins.io] Move controller (VM) to AWS
      • VM is up, running and available
      • Next step: fine tune configuration adn start working on EC2 VM agents
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • We can resume working on this
      • Linux agents are available, GC almost finished
      • Need to work on Windows (Damien)
      • Puppet config for ec2 cloud agents (Jay and then Damien)
    • On hold (Stephane is in sick days)
      • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
      • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart

  • [packer-images] track Linux base images with updatecli

    • Linux base images for Amazon AMI are tracked
    • Windows base images for Azure
    • Added the Docker Linux base images \o/
    • Next step: Azure Linux images
      • Need to sync up (Jay ↔ Damien)

  • Infra stats missing October 2024 data for stats.jenkins.io Plugin Installation Trend feature

  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org

    • On hold, missing time (UC/ci.jio)

  • plugin stats for bouncycastle-api are missing data.

    • On hold: need help to fix

• New Issues (to triage):

  • In the new milestone:
    • Remove BlueOcean from all of our controllers
    • [cert.ci.jenkins.io] Keep Maven tools up to date with updatecli
    • [updatecli] Track GitHub SSH in bound IPs allowed from our controllers and agents
    • External user struggling to submit story to stories.jenkins.io
    • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • Keep in triage:
    • Add monitoring for CD secrets updates
    • [ci.jenkins.io] Set up an ECR pull through cache
    • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
    • [ci.jenkins.io] Move ephemeral Linux containers to AWS
    • Switch agent (java home) to JDK21 default
    • Switch default JDK to 21 for pipeline libraries
    • Switch default JDK to 21 for build tools
    • Move controllers to JDK21 (runtime)
    • Move agents to JDK21 (runtime)
    • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2024-11-26 Milestone · GitHub)