Infrastructure Team Meeting - April 16, 2024

[dduportal (Damien Duportal)]

2024 04 16 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: 2.454 is out
   • Release process was
   • Changelog has been merged!
   • Usual Packaging stuck on OSUOSL => need infra action, Damien to open an issue to track this, but AFTER the LTS (see below)
   • Docker image is being built
2. Tomorrow is LTS day! Don’t break anything
3. Java patch day today: expect PRs to upgrade JDKs soon!

Cloud Billing

• No cloud billing update this week (Damien’s late)
• Azure bill is $100 higher than what the cost explorer shows => this the cost of monthly “standard” support

Upcoming Calendar

• Next Weekly: 2.455 the 23 April 2024

• Next LTS: tomorrow (17 April 2024) 2.440.3

• Next Security Release as per jenkinsci-advisories: N.A.

• Next major event: N.A.

• Done:

  • [ci.jenkins.io– Startup and build failures with java.lang.NoClassDefFoundError: org/jenkinsci/plugins/gitclient/RepositoryCallbackerror in the datadog plugin
  • [accounts.jenkins.io] Delete account angelvalangelov
  • Delete account accounts.jenkins.io

• Closed as not planned:

  • Requesting permission for “repository-permissions-updater” repository
  • Extend CloudSet for spotinst-plugin
  • Pipeline starts every time one of the nodes boots up
  • Weekly repo issue
  • Delete Jira TEST project

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Nothing done: Hervé is out and Stéphane and Damien were busy

  • Linux arm64

    • infra.ci.jenkins.io on arm64 (controller and agents)
      • Controller is now running on arm64 (along with all agents except a few needed to run on x86_64)
        • Dedicated node pool and subnets (more isolation)
        • Faster unmount/remount (at the cost of init of empty volume to require manual operation)
      • To do before closing:
        • Cleanup of the PV/PVC/disk/snapshots leftovers (resource manually created during operation)
          • Thursday (after LTS)?
        • Issue opened about using a different storage kind (pricing/performance) to decrease cost
          • And remove the PVC’s spec.datasource unused directive
        • Issue to open: The (Jenkins) agent label linux-arm64 is used for both VM and container. That makes the execution of jobs undeterministic in behavior and execution time
    • Migrate release.ci.jenkins.io to arm64 (controller only)
      • Controller is now running on arm64
        • Dedicated node pool and subnets (more isolation)
        • Weekly release today went well on this one
        • Faster unmount/remount (at the cost of init of empty volume to require manual operation)
      • No agent will be run on arm64!
      • To do before closing:
        • Set up subnet NSG to isolate release.ci (like infra.ci) (issue to add)
        • Cleanup of the PV/PVC/disk/snapshots leftovers (resource manually created during operation) => same operation as infra.ci and weekly.ci cleanups (one shot)
        • Issue opened about using a different storage kind (pricing/performance) to decrease cost
    • Cost on privatek8s cluster
      • Expect increase costs: linuxpool still runs some workload. We want to keep this cost increase because we want isolation between controllers and also other services
    • Next arm64 steps:
      • docker-builder (“all in one” packer image to be used + arm64)
      • LDAP, keycloak migration
      • Ping mirrorbits maintainers

  • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts

    • Nothing done, need to send an email to the developers mailing list

  • Release of jenkins/* container images on DockerHub errors with HTTP/429

    • We now have spread the outbound requests through 3 IPs (instead of 1)
      • Worked well with docker-agent releases yesterday
      • Worked with weekly Core image today
    • Next step, SIG platform actions:
      • Decrease amout of pulled layers by stop uisng Temurin container images to get the installer (and use the binary installer instead)
        • WiP on controller => Bruno
        • WiP on agents => Hervé
    • Keeping the issue opened to track this part

  • Move the jenkins-stats repositories to the jenkins-infra organization

    • Nothing done (yet), need to resume

  • migrate plugin-site-api generate-data from ci.jenkins.io to infra.ci.jenkins.io

    • Nothing done (yet), need to resume

  • New Mirrors

    • New Jenkins mirror in Romania by RCS&RDS
    • New Jenkins mirror in Romania by Hostico
    • One has been pinged (missing HTTPS but maintainer was in holidays)
    • Other mirror: need to check FTP credentials

  • [AWS] Migrate ci.jenkins.io EKS clusters from CloudBees AWS account to Jenkins AWS (sponsored) account

    • WiP on bootstrap (terraform for IMA model)
    • Next steps:
      • Perform effective bootstrap in pair
      • Start working on a new “cik8s” cluster for ci.jenkins.io

• ToDo (next milestone) (infra-team-sync-2024-04-23 Milestone · GitHub)

  • Let’s Encrypt Azure Service Principal passwords expired for cert.ci.jenkins.io and trusted.ci.jenkins.io
    • Stéphane
  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release
  • release.ci JobDSL
    • Damien to open an issue!