Infrastructure Team Meeting - November 05, 2024

[dduportal (Damien Duportal)]

2024 11 05 Jenkins Infra Meeting

Attendees

• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Releases:
   • Last Week: 2.483 went well
   • Last Week: 2.479.1 LTS went well
   • This Week: 2.484 started on time at 10:30 UTC

Upcoming Calendar

• Next Weekly: 2.485
• Next LTS: 2.479.2 (27 Nov 2024) - Kris Stern is release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • trusted.ci Azure VM credential (20 Nov.)
  • release core Azure Vault credential (20 Nov.)
• Next major event:
  • Jenkins Elections - voting start 1st of November
  • 2 weeks “Releases break” at the end of calendar year
    • Delays LTS 2.479.3 to 8 Jan 2025
  • Jenkins Contributor summit - 31 January 2025 in Brussels, Belgium
  • FOSDEM - 1 and 2 February - Brussels, Belgium

Cloud Budgets

• Azure CDF:
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $4.1k
  • November: $459 (forecast at 3.4k)
• Azure Sponsorship (Microsoft Credits) - Remaining: $33039 ($66961 consumed) until May 2025
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $12.9k consumed
  • November: $985 (Forecast at ~8k)
• DigitalOcean - Remaining $15,638 (~5k consumed) until 02 January 2025
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $195,67 (invoice)
  • November: $20 (Forecast at ~$150)
• AWS:
  • CloudBees:
    • August: $6.3k
    • September: $6.1k
    • October: $6.4k
    • November: $715 (forecast at $6k)
  • Sponsored account
    • October: $178
    • November: $28 (forecast at ~$210)

Notes

• Done:

  • Update Center:
    • [Azure Update Center] Serve directory listing instead of redirecting to mirrors (ending in HTTP/404)
    • [cloudflare] Collect access logs in a location where we can view and request them
  • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.479.1
  • Artifact Caching proxy issue (Pipeline can’t find the remote artifactory)
    • Plugin contributor did break their pom.xml but we helped them fix it (by… rollbacking their change of repository.id)
  • Adding GSoC project to jenkins-infra

• Closed as not planned:

  • Thanks @timja for helping plugin contributor in:
    • CD pipeline for railflow-testrail plugin fails to upload to maven repo
    • CD pipeline fails to upload to Artifactory for some users

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • We need a 6th brownout!
      • Proposal: Thursday 7 Nov. 09:00am UTC → Friday 8 Nov. 09:00am UTC
        • Need blog post, status.jio ASAP
    • To do before production
      • Conclude that the 6th brownout shows no red flag (5th brownout did not)
      • Add a mirrorbits refresh step in the update_center2 publish script before scanning mirrors
        • Warning: time impact?
        • Alternative: decrease the refresh repo frequency to 10 min (today) to 3-4 min (based on the required time)
      • Add archives.jenkins.io as a 3rd mirror (send data to it, and add it as a mirror) with a lower weight
        • Goal: add an alternative to Cloudflare
        • Risk: do not consume too much DigitalOcean billing on it
        • Using as a fallback instead of a mirror? Depends on the result of mirrorbits refresh
    • Proposed production date: Monday 18 November at 09:00am UTC
  • [packer-images] track Linux base images with updatecli
    • Delayed as Jay is off
  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Define virtual networking for AWS
      • WiP on Network ACLs by Damien to secure the controller VM subnet
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • WiP on IAM roles and permissions to ensure Terraform is not allowed to manage permissions itself
    • [ci.jenkins.io] Move controller (VM) to AWS
      • Delayed until Network is fixed (next step: create empty VM)
  • Multiple requests for GSoC 2024 Plugin Modernizer Tool
    • Valentin and Tim should transfer repository this week: waiting for them before setting up CI/CD on this project
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
      • Delayed: down prioritized due to AWS work

• New Issues (to triage): DELAYED

  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • [terraform-aws-sponsorship] updatecli manifest for outbound ips
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • [ci.jenkins.io] Move ephemeral Linux containers to AWS
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)