Infrastructure Team Meeting - March 19, 2024

[dduportal (Damien Duportal)]

2024 03 19 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly (2.450)
   • WAR and Docker images are out => go for infra.ci delivery
   • Other packages are being synced to mirrors (max 1 hour propagation, see point 2. below)
   • Changelog is out!
2. Reminder that OSUSL mirrors are down (1 of the 2) and it might have impact for tomorrow’s LTS packaging phase (TL;DR; we’ll have to wait 1 hour for propagation)

Upcoming Calendar

• Next Weekly: 2.451 the 26 March 2024
• Next LTS: 2.440.2 tomorrow (20 March 2024), Kris Stern is release lead and is on track
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • cdCon, April 16-18 in Seattle: Mark and Deeraj will be there

Notes

• Done:

  • plugin-site build is failing
    • Due to an agent template change on infra.ci.jenkins.io last week, deployment started to fail
    • Root cause: the new agents (Linux arm64 VM instead of x86 pod templates) weren’t allowed to access the production file share through their network
    • Fixed and green
    • Post-mortem: monitor these jobs using reports.jenkins.io with information that could be public (job name, status and last date) combined with datadog probe to detect last build time and status to alert us if failing or not run since X minutes/ Y hours. See the “monitor private instances” task below
  • SSL certificate on ci.jenkins.io and assets.ci.jenkins.io expires 28 Mar 2024
    • Root cause: as each VM migration, our old puppet module for Letsencrypt creates a new account and fail because only 1 is needed
    • Fixed it manually (with details on the issue)
    • Post-mortem: monitoring certificate ages with datadog/pagerduty (instead of waiting for Mark)
  • Delete Jira VERTEST project
  • Permission setup for scm-filter-aged-refs-plugin strange
  • Not able to get jenkins password
  • Upgrade to Kubernetes 1.27
    • MISSING THE LOGO !!!
    • The 2 AKS clusters were done yesterday and today (before the weekly)
    • We took the opportunity to bump falco (security) and nginx (ingress) charts as part of the operation
    • Kubernetes 1.28 issue + planning for next week: @smerle
  • Mirror in Singapore
    • Close to Servana: spreading the workload in Asia between both
    • Note: to add in the Sponsors page on jenkins.io (Jenkins Board takes care of it: action item for them)

• Closed as not planned:

  • Synopsys Security Scan Plugin version 1.3.0 has not been published yet in Jenkins marketplace
  • Requesting latest Plugin Health Scoring application logs
  • Getting my Jenkins account
  • Can not login success
  • unable to login to page with the credentials used to create
  • Bridge IRC channels to matrix/gitter

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • update_center2 is now running every 5 minutes (instead of 3) and updates the metadatas to both current UC and the new mirror system (Azure + Cloudflare storages + mirrorbits scan on eac)
    • crawler is also updating everything (tool install metadata)
    • Next steps:
      • Start testing the new mirror system
      • Performance benchmark of the new system
      • Work on the JEP on the cloudflare fallbacks
      • Plan for a brownout

  • Check if we could replace blobxfer by azcopy

    • WiP on the jenkins-infra/mirror-scripts
      • Waiting for after the LTS release
      • Puppet, and digging history…
    • Next steps:
      • Once mirror-scripts are done: azcopy is everywhere
      • Cleanup: removal of blobxfer and storage accounts clean up

  • infra.ci.jenkins.io on arm64 (controller and agents)

    • WiP: docker-webbuilder, agent image used by both ci.jenkins.io and infra.ci (node, ruby, yarn, etc.)
      • Using VMs on most of the infra.ci with the new all in one images
      • ci.jenkins.io additional work (labels, arm64 node pool on cik8s cluster, etc.)
      • missing tool in plugin-site (typos and typos-checkstyle)
    • Next steps:
      • Agents “JNLP-linux”
      • Clean up docker-hashicorp-tools on infra.ci.jenkins.io

  • To host versioned jenkins.io docs on docs.jenkins.io

    • No advances, let’s continue this milestone

  • Service Principal used by infra.ci.jenkins.io to spawn Azure agents expires on 2024-03-22

    • Azure SP used by infra.ci has its password expiring this Thursday.
    • It’s used to spin up VM agents on Azure
    • Todo: generate new credential, update secrets, deploy, test and profit

  • jenkins.io and plugin-site-issues.jenkins.io are misbehaving on IPv6 with a low MTU

    • Should be closed end of week unless user has actionnable for us. Alas we cannot tune MTU on Microsoft networks.

  • Wrong email on registration to Jenkins Jira

    • Wait for user response, or close next week if none

  • New Jenkins mirror in Romania by Hostico

    • Waiting for them for final actions

  • ACPs

    • Artifact Caching Proxy (ACP) does not cache artifacts from Maven Central
    • Artifacts proxy does not deliver incremental jars
    • Unexpected delays building small plugin on linux agent
    • Let’s digest this for 24h and decide who want to lead

  • [plugin-health-scoring] Configure new job on ci.jenkins.io

    • To be done

  • [AWS] Migrate ci.jenkins.io EKS clusters from CloudBees AWS account to Jenkins AWS (sponsored) account

    • WiP: boostrap admins on the new AWS account
    • Next step: decide who drive the creation of the ci.jenkins.io EKS cluster in it

  • The UpdateCli step fails regularly when processing jenkins.io PRs

    • Not a blocker (even if annoying)
    • back to backlog for now (AWS is the priority)

  • Add a new private kubernetes cluster in the new sponsored azure subscription

    • target was infra.ci pod agents
    • back to backlog for now (AWS is the priority)

  • Revoke an OpenVPN cert for NotMyFault

    • Waiting after the LTS to deploy
    • Might have impact on older VPN certificates

• ToDo (next milestone) (infra-team-sync-2024-03-26 Milestone · GitHub)

  • Quietly skipped incrementals deployment after retry · Issue #4002 · jenkins-infra/helpdesk · GitHub
  • Restrict issue creation in `MEETING` component · Issue #3996 · jenkins-infra/helpdesk · GitHub
  • Permissions for Jenkins Security Scan repos · Issue #3993 · jenkins-infra/helpdesk · GitHub