Infrastructure Team Meeting - October 22, 2024

[dduportal (Damien Duportal)]

2024 10 22 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.481 went smoothly!
   • This Week: 2.482 started on time and is progressing as expected
2. Jenkins elections are in progress
   • Voter registration is open until 31 Oct 2024, 71 voters registered
   • Candidate statements are available
3. Meeting time change coming due to end of summer time
   • Is it ok? OK with all attendees

Upcoming Calendar

• Next Weekly: 2.483
• Next LTS: 2.479.1 30 Oct 2024 - Mark Waite release lead, checklist
  • End of life for Java 11 \o/
  • Jetty 12 upgrade from Jetty 10
  • Spring Security 6 upgrade from Spring Security 5
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks): N.A.
• Next major event:
  • FOSDEM, Feb 1-2 2025 in Brussels, Belgium
• Upcoming brownout (5th) for the new Update Center from Wednesday 23, 14:00 UTC, for 24 hours

Cloud Budgets

• Azure (CDF paid)

  • July: $4,571 (invoice)
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $2,841 (forecast at 4.2k)
    • Increased publick8s outbound bandwidth (+$8 daily)
    • Increased DNS cost (public queries) (+$7 daily)

• Azure Sponsorship (Microsoft Credits) - Remaining: $38828 ($61172 consumed) until May 2025

  • July: $10k consumed
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $8.1k (Forecast at ~11k)

• DigitalOcean - Remaining $xxx (~4k consumed) until 02 January 2025

  • July: $176.01 (invoice)
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $137 (Forecast at ~$185)
    • slight outbound increase (archives.jenkins.io usage)

• AWS:

  • CloudBees:
    • July: $6.5k
    • August: $6.3k
    • September: $6.1k
    • October: $4.6 (forecast at ~6.7k)
  • Sponsored account - $xxxx left
    • October: $69 (forecast at ~$100)

Notes

• Done:

  • Older Debian package availability is incorrect on get.jenkins.io

  • ci.jenkins.io Azure credential

    • [ci.jenkins.io] Service Principal used by ci.jenkins.io to spawn Azure agents expires on 2024-10-19
    • Docker agents are not launched on ci.jenkins.io

  • [ci.jenkins.io] project jenkins-infra/docker-plugin-site-issues is not built on ci.jenkins.io (scan permission errors) and has a different pipeline than infra.ci

  • [Update Center] generate HTML pages with absolute links

  • (packer-images): Add Linux AMD64 and ARM64 AMIs For EC2 agents

  • SPAM-related issues

    • Block user marilynacrabtree for Jira issue spam
    • Block user emilytrantow due to Jira issue spam

• Work in Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • [Update Center - Azure] HTTP/404 responses instead of redirections with error AH01630: client denied by server configuration
    • Potential UC publication to optimize

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)

    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • ci.jio IAM user created with permissions => @jay ready to test EC2 plugin locally
      • (packer-images): Add Garbage Collector for AWS
    • [ci.jenkins.io] Define virtual networking for AWS
      • On hold (missing Security groups at least)
    • [ci.jenkins.io] Move controller (VM) to AWS
      • Started to write a Terraform module for this

  • Updatecli tasks

    • [terraform-aws-sponsorship] updatecli manifest for AWS terraform provider
    • [terraform-aws-sponsorship] updatecli manifest for aws/vpc module

  • infra-statistics has no new data since July 1st 2024

    • KK and Andrew are working on updating stats.

  • GSoC: all put on hold

    • Multiple requests for GSoC 2024 Plugin Modernizer Tool
    • Adding GSoC project to jenkins-infra
    • RPU GSoC project with terraform integration on infra’s backend

• New Issues (to triage):

  • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)

    • Drafting the permissiosn and RBAC needs
    • [ci.jenkins.io] Move ephemeral Linux containers to AWS

  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart]([get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading `mirrorbits` chart · Issue #4240 · jenkins-infra/helpdesk · GitHub)

    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
    • Validated the PoC of the Docker Image with Azure technical credentials
    • Publication of Docker image (repo, build, DockerHub)
    • Next step: helm chart

  • Use Docker Terraform Provider to manage Docker organisation

  • TODO: create issue for October JDK updates

• Delaying triage:

  • [terraform-aws-sponsorship] updatecli manifest for outbound ips
  • [ci.jenkins.io] Set up an ECR pull through cache
  • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
  • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
    • packer IAM user know has the proper permissions
  • chocolatey installer not working in packer-images
  • Switch agent (java home) to JDK21 default
  • Switch default JDK to 21 for pipeline libraries
  • Switch default JDK to 21 for build tools
  • Move controllers to JDK21 (runtime)
  • Move agents to JDK21 (runtime)
  • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)

• ToDo (next milestone) (infra-team-sync-2024-10-29 Milestone · GitHub)