Infrastructure Team Meeting - October 01, 2024

[dduportal (Damien Duportal)]

2024 10 01 Jenkins Infra Meeting

Attendees

• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)
• @jayfranco999 (Jay Reddy)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.478 was successfully delivered
   • This Week: 2.479 delayed to tomorrow for security release

Upcoming Calendar

• Next Weekly: 2024-10-8 (2.480)
• Next LTS: tomorrow 2024-10-02 2.462.3 - Kris Stern release lead
• Next Security Release as per jenkinsci-advisories: tomorrow 2024-10-02
• Next major event:
  • FOSDEM 2025 - Brussels - Feb 1-2, 2025
  • jenkins contributor summit - Brussels - 31 January 2025

Cloud Budgets

• Azure (CDF paid)

  • June: $4,287 (invoice)
  • July: $4,571 (invoice)
  • August: $4,552 ($4,452 cost + $100 monthly support)
  • September: $3,811 consumed
  • October: $19 (Forecast $3979 but too early)

• Azure Sponsorship (Microsoft Credits) - Remaining: $49166 ($50834 consumed) until May 2025

  • June: $7.3k consumed
  • July: $10k consumed
  • August: $10.5k consumed
  • September: $8268 (Forecast at ~10.3k) not updated since 2024-09-24 only damien have access
    • ci.jenkins.io
    • Redis databases

• DigitalOcean - Remaining $15,823.54 (~4k consumed) until 02 January 2025

  • June: $165.32 (invoice)
  • July: $176.01 (invoice)
  • August $200.08 (invoice)
  • September: $158.84 (invoice) [was not yet edited during the video recording hence my error]
  • October: nothing yet

• AWS:

  • CloudBees:
    • June: $5,862
    • July: $6.5k
    • August: $6.3k
    • September: $6.1k
    • October: too early
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Started using it

• To open an issue describing the “cloud billing plans” for the upcoming 4 months

  • Moving ci.jenkins.io out from Azure (credits) to AWS (to consume credits in AWS and allow Azure Credits to be spent until May) : [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship) · Issue #4313 · jenkins-infra/helpdesk · GitHub
    • Controller is easy to move
    • Agent VMs is easy (packer, then EC2 plugin) both Linux (Intel, ARM64) and Windows are supported
    • Linux container agent is easy => EKS (includes internal ACP)
    • !! Windows container agents: ACI → Kubernetes
    • !! Docker Registry: does AWS ECR support mirroring mode
  • Move CloudBees AWS remaining VMs to DigitalOcean (to consume credits):

Notes

• Done:

  • JIRA spam:
    • Block sunwindigital for Jira issue spam
    • Delete user ryanhiggs1 for Jira issue spam
    • Block user hitclubinstitute for Jira issue spam
    • Block user dora45rose for Jira issue spam
  • [Terraform] Expiration of credentials for Backend States and Cloudflare API
  • Set up the AWS (sponsored) account as a Terraform project
  • [ci.jenkins.io] Ensure our build agents have enough entropy (rngd, haveged, etc.)

• Closed as not planned:

  • Plugin trends stats page is missing all plugins whose artifactId comes after dry when sorting case sensitively

• Work in Progress:

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Define virtual networking for AWS
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
  • [cert.ci.jenkins.io, trusted.ci.jenkins.io] Azure Principal used for Let’s Encrypt DNS challenges expires on 2024-10-06
  • infra-statistics has no new data since July 1st 2024
  • Gradle plugin uses a proprietary dependency
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
  • To host stats.jenkins.io GSoC 2024 project in jenkins-infra
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

• New Issues (to triage):

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Set up an ECR pull through cache
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
    • [ci.jenkins.io] Move controller (VM) to AWS
  • Upgrade Maven to 3.9.6 or newer for GitHub Actions
  • Build errors of docs.jenkins.io PR’s appear to be infra-related
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)