Infrastructure Team Meeting - October 15, 2024

[dduportal (Damien Duportal)]

2024 10 15 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @jayfranco999 (Jay Reddy)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Jenkins Weekly Releases
   • Last Week: 2.480 went smoothly!
   • This Week: 2.481 started on time
2. Jenkins project has received a grant from Alpha Omega
   • CSP project funding Shlomo Dahan and Yaroslav Afenkin
   • Announced in a blog post, second blog post once Yaroslav signs his contract
3. “Software of Public Interest”, former home of the Jenkins Project, has a sitting $9k for us.
   • Not easy to spend, cannot transfer
   • So we’ll use it to fund contributors travels to FOSDEM \o/
4. Jenkins elections are in progress
   • Voter registration is open until 31 Oct 2024
     • 70 registered to vote
   • Candidate statements are available

Upcoming Calendar

• Next Weekly: 2.482
• Next LTS: 2.479.1 30 Oct 2024 - Mark Waite release lead, checklist
  • End of life for Java 11 \o/
  • Jetty 12 upgrade from Jetty 10
  • Spring Security 6 upgrade from Spring Security 5
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • ci.jenkins.io Azure VM agents - [ci.jenkins.io] Service Principal used by ci.jenkins.io to spawn Azure agents expires on `2024-10-19` · Issue #4358 · jenkins-infra/helpdesk · GitHub
  • Need to check in Azure UI with @smerle and @dduportal
• Next major event:
  • FOSDEM, Feb 1-2 2025 in Brussels, Belgium
• Meeting time change coming due to end of summer time
  • Proposal: Keep same meeting time in Paris time?
    • Proposal accepted by all in the meeting

Cloud Budgets

• Azure (CDF paid)

  • July: $4,571 (invoice)
  • August: $4,552 (invoice)
  • September: $3,810 (invoice)
  • October: $1,913 (forecast at 4.1k)

• Azure Sponsorship (Microsoft Credits) - Remaining: $41006 ($58994 consumed) until May 2025

  • July: $10k consumed
  • August: $10.5k consumed
  • September: $10.4k consumed
  • October: $5.9k (Forecast at ~12.3k !!!)

• DigitalOcean - Remaining $15,852.91 (~4k consumed) until 02 January 2025

  • July: $176.01 (invoice)
  • August $200.08 (invoice)
  • September: $158 (invoice)
  • October: $85 (Forecast at ~$170)

• AWS:

  • CloudBees:
    • July: $6.5k
    • August: $6.3k
    • September: $6.1k
    • October: $3.2k (forecast at ~6.8k)
  • Sponsored account - $59,966 left
    • October: $34 (no forecast yet)

• Mark and Damien propose a 2025 cloud budget to CDF

Notes

• Done:

  • Credential Rotations:

    • [infra.ci.jenkins.io] Credential rotation for deployment of plugins.jenkins.io and docs.jenkins.io
    • [trusted.ci.jenkins.io] Credential rotation of Azure Files for www.jenkins.io and javadoc.jenkins.io

  • Spam:

    • Block user katty867 due to Jira issue spam
    • Block user lazyunicorn for Jira issue spam
    • Block user leooscarees@gmail.com due to Jira issue spam
    • Block user irischamplin due to spam Jira comments
    • Block user coxex96560 for Jira issue spam
    • Suspicious user eleanorladd on JIRA

  • Backend Extension indexer:

    • Big shoutout to @daniel-beck for solving these problems
    • extension-indexer runs with wrong Java (11 instead of 17) on infra.ci
    • Credential artifact-caching-proxy-credentials unavailable on ci.jenkins.io

  • Outages/Infra issues:

    • kubernetes infracijenkinsio-agents-1 arm64 nodepool la64n14agt1 not scaling
    • Packer-Images failure on windows 2019 due to ssh-server start failing

  • Unable to Create a Jira Account

  • Request for GH Personal Access Token for plugin project.

  • No documentation and no javadoc shown on my plugin page

  • Plugin site improperly displays GH releases with empty title

    • Thanks @alecharp for the fix, @daniel-beck, @smerle and @zbynek for the support

  • Upgrade Maven to 3.9.6 or newer for GitHub Actions

  • Build errors of docs.jenkins.io PR’s appear to be infra-related

  • [Plugin Health Score] Scores not computed - Getting logs from plugin-health.jenkins.io

    • Scores are now computed since a few days, thanks to @alecharp and @smerle
    • We have to close status.jenkins.io outage and ready to close

• Closed as not planned:

  • Not able to login to jenkins forgot username and password
  • Request for Admin Access to the GitHub Repository of the Environment Variables Status Sync Plugin
  • ESIC command execution issue
  • New Docker based Jenkins installation is failing to download plugins

• Work in Progress:

  • [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
    • [ci.jenkins.io] Move ephemeral VM agents to AWS
      • WiP on packer-images to create Linux AMIs for ephemeral agents (using EC2) + GC of AWS resources
      • Next: Windows AMIs and optimizations (security, etc.)
      • Start adding EC2 plugin on ci.jenkins.io
    • [ci.jenkins.io] Define virtual networking for AWS
      • packer is out of this scope
      • WiP on creating the VPC, subnets, gateways, etc.
        • Using same terraform module as in the past
        • The module only support, today, 1 public egress IP per gateway
        • But there is an opened PR (from this summer): let’s start with 1 public IP per subnet (module’s default): 1 for controller, 1 for VM agents and 1 for container agents
        • Another discovery: we need a public subnet for hosting the NAT gateway to allow internet egress (while resources will stay in private subnet)
      • Looks good: incoming optim. on security groups, but foundation is good for starting the VMs within
  • infra-statistics has no new data since July 1st 2024
  • [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
    • [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
      • Image and script are readey to roll
      • WiP on the helm chart and setting up Azure SP for credentials
    • Delaying as Stephane is on day off

• ToDo (next milestone) (infra-team-sync-2024-10-22 Milestone · GitHub)

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • [Update Center - Azure] HTTP/404 responses instead of redirections with error AH01630: client denied by server configuration
    • [Update Center] generate HTML pages with absolute links
    • Eventually get away from cloudflare, or at least find 1 VM with rsync somewhere
  • GSoc related:
    • RPU GSoC project with terraform integration on infra’s backend
    • Adding GSoC project to jenkins-infra
    • Multiple requests for GSoC 2024 Plugin Modernizer Tool
  • Providing NodeJS to developers:
    • Build errors of docs.jenkins.io PR’s appear to be infra-related
    • Packer: Caching of the installer? => issue to open
      • Also for python: chocolatey installer not working in packer-images
  • [infra.ci.jenkins.io](Builds stucks due to GH API rate limit]([infra.ci.jenkins.io] Builds stucks due to GH API rate limit · Issue #4165 · jenkins-infra/helpdesk · GitHub)
    • Back to backlog
  • @smerle need admin. access to the GH jenkins-infra organization
    • @dduportal to ask on the proper mailing list

• New Issues (to triage):

  • GitHub Markdown adminitions are not supported by plugins.jenkins.io

    • Moving it to plugin-site issue tracker as it’s out of our scope (frontend dev?)

  • Updatecli manifests to add/update to new milestone:

    • updatecli manifest for outbound ips
    • updatecli manifest for AWS terraform provider
    • updatecli manifest for aws/vpc module

  • ci.jenkins.io on AWS:

    • Ephemeral VMs → Packer images subtasks
      • (packer-images): Add Linux AMD64 and ARM64 AMIs For EC2 agents
        • Adding to new milestone
      • (packer-images): Add Garbage Collector for AWS
        • Adding to new milestone
      • (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
        • Keep in triage until we can start it
    • [ci.jenkins.io] Move controller (VM) to AWS
      • Adding to new milestone as the network looks ready
    • [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
      • Keep in triage
    • [ci.jenkins.io] Move ephemeral Linux containers to AWS
      • Keep in triage
    • [ci.jenkins.io] Set up an ECR pull through cache
      • Keep in triage
    • [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
      • Keep in triage

  • Infras outages/issues:

    • Older Debian package availability is incorrect on get.jenkins.io
      • Need to repopulate Redis data from scratch to fix the issue as it is a remnant of the redis migration
      • Let’s check the sizing of archives.jenkins.io befiore the operation
      • Targeting Wednedsday
      • Adding to new milestone
    • [ci.jenkins.io] project jenkins-infra/docker-plugin-site-issues is not built on ci.jenkins.io (scan permission errors) and has a different pipeline than infra.ci
      • Adding to new milestone
    • chocolatey installer not working in packer-images
      • Keep in triage
    • Nobody noticed that the last successful extension-indexer run was in August 2023
      • Keep in triage

  • Delaying triage:

    • Switch agent (java home) to JDK21 default
    • Switch default JDK to 21 for pipeline libraries
    • Switch default JDK to 21 for build tools
    • Move controllers to JDK21 (runtime)
    • Move agents to JDK21 (runtime)
    • [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)