Infrastructure Team Meeting - November 12, 2024

[dduportal (Damien Duportal)]

2024 11 12 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.484 went well
   • This Week: 2.485 (started on time at 10:30 UTC)

Upcoming Calendar

• Next Weekly: 2.486
• Next LTS: 2.479.2 (27 Nov 2024) - Kris Stern is release lead
  • RC tomorrow (13 Nov. 2024)
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • trusted.ci Azure VM credential (20 Nov.)
  • release core Azure Vault credential (20 Nov.)
• Next major event:
  • Jenkins Elections - voting started 1st of November, registration reopened
  • 2 weeks “Releases break” at the end of calendar year
    • Delays LTS 2.479.3 to 8 Jan 2025
  • Jenkins Contributor summit - 31 January 2025 in Brussels, Belgium
  • FOSDEM - 1 and 2 February - Brussels, Belgium

Cloud Budgets

• Azure CDF:
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $4.0k (invoice)
  • November: $1.5k (forecast at $4.1k)
• Azure Sponsorship (Microsoft Credits) - Remaining: $30319 ($69681 consumed) until May 2025
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $12.9k consumed
  • November: $3.7k (Forecast at $9.1k)
• DigitalOcean - Remaining $15,559 ($4k consumed) until 02 January 2025
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $195.67 (invoice)
  • November: $65 (Forecast at $165)
    • Requesting extension in Dec.
• AWS:
  • CloudBees:
    • August: $6.3k
    • September: $6.1k
    • October: $6.4k
    • November: $2.3k (forecast at $6.2k)
  • Sponsored account
    • October: $178
    • November: $135 (forecast at $500)

Notes

• Done:

  • [Azure Update Center] HTTP/404 on directories which are symlinks
  • infra-statistics has no new data since July 1st 2024
  • Azcopy 10.27.0 errors:
    • stats.jenkins.io: deployment fails due to files with the same but different cases
    • [Plugin page not found https://plugins.jenkins.io/markdown-params/ ](Plugin page not found [https://plugins.jenkins.io/markdown-params/](markdown-params) · Issue #4379 · jenkins-infra/helpdesk · GitHub)
    • Javadoc fails to deploy (trusted.ci.jenkins.io)
  • BOM release impossible due to missing Jenkins check
  • 401 during CD deployment
  • trusted.ci.jenkins.io: remove all AWS plugins

• Closed as not planned:

  • Jenkins deadlock Error

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • 6th brownout was the last (and successful)
    • Fixed all unexpected HTTP/404 and Cloudflare access logs are streamed and parsed in datadog
      • <…mirrors.updates.jenkins.io…>//latest/<…> => user agent: curl => this is an error and expected to be one
    • Production: Monday 18 Nov. 2024
      • Blog post + announcements to do
    • Optimizations
      • Add archives.jio as fallback
      • Make update_center2 generation a bit faster (room for optimization)
        • Cron is set to 6 min
      • Minor: Update Apache htdocs as last step in update_center2
      • Cleanup:
        • Old VM: pkg.origin.jenkins.io to be migrated 1st (so no cleanup to run on the VM itself)
        • update_center2 cleanup of the old VM => 1 month after the production (if no rollback)
        • DNS aws.updates.jenkins.io + Apache Vhost in Puppet for the VM
  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Define virtual networking for AWS
      • EKS require at least 2 AZs (and 2 subnets as such) => @smerle
      • Network ACL + Security Groups for VMs => @dduportal
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • Permissions problem fixed by using naming
      • Requires 2 subnets on distinct AZs, but we can use only 1 AZ
      • WiP by @smerle
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • (packer-images): Add Garbage Collector for AWS => @jayreddy
    • [ci.jenkins.io] Move controller (VM) to AWS
      • Blocked by Network
  • Updatecli tasks
    • [packer-images] track Linux base images with updatecli => @jayreddy
      • AWS base Ubuntu images are pinned and tracked
      • Azure Windows: same
      • WiP on Docker Linux based images (pinning and tracking)
      • WiP Azure Linux (pinning and tracking)
    • [terraform-aws-sponsorship] updatecli manifest for outbound ips => @jayreddy
      • WiP:
        • HCL parser in updatecli has an issue updating complex HCL “locals” structure
        • VPN outbound IP is now exported in our automatic reports
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
      • Delayed with the “looong French week end”
  • ci.jenkins.io: upgrade datadog plugin from 7.x to 8.x
    • Planning for Thursday
    • Need to announce (status, mail, IRC)
  • Multiple requests for GSoC 2024 Plugin Modernizer Tool
    • Nothing done: need to ping Valentin
  • RPU GSoC project with terraform integration on infra’s backend
    • Blocked (nothing done)
    • Should be only “documenting” things

• New Issues (to triage):

  • Maven Central artifacts are being downloaded from repo.jenkins-ci.org
    • Artifactory side configuration (change? hiccup? surprise ?)
  • Add monitoring for CD secrets updates
  • Monitor private controller jobs
    • Each “important” pipeline (RPU, website deployment, etc.) will call a pipeline library to write to reports.jenkins.io (name, status, timestamp)
    • Datadog monitors the reports and alert us in case of failure or max. elapsed time since last update
    • @dduportal start issue / writing specs. Then @jay will get to learn pipeline libraries

• Delayed issues:

  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • [ci.jenkins.io] Move ephemeral Linux containers to AWS
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2024-11-19 Milestone · GitHub)