Infrastructure Team Meeting - September 12, 2023

[dduportal (Damien Duportal)]

2023 09 12 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)

• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @ash-sxn Ashutosh Saxena
• @kB_yaTxnRYiRgiB6BgSZvA (Oleg Nenashev)

Announcements

1. Weekly:
   • 2.423 is released: packages, Docker image are out
   • Last release items to be done later today
   • Jenkins Infra team is ready to deliver this version to infra.ci/weekly.ci

Upcoming Calendar

• Next Weekly: 2.424 of Jenkins, the Septembre 19 2023
• Next LTS: 2.414.2 September 20, 2023
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • DevOps World tour is coming (to meet a community member for each of these events during “Let’s talk about Jenkins”)
    • NYC: Sept. 13/14, 2023
    • Chicago: Sept. 27
    • Santa Clara in Oct.
    • London end of November 2023
  • FOSDEM
    • 3 & 4 February 2024

Notes

• Ashutosh Saxena for GSoc

  • Until now, used GitHub Actions + Docker Hub Registry for Docker image
  • ghcr.io would be a good place to start, because any maintainer could participate without special credentials => let’s move to ghcr
  • jenkins (jenkins/jenkins:lts-jdk17) and jenkinsciinfra (jenkinsciinfra/ldap:latest)
    • Mapped 1:1 to GitHub organization: jenkinsci, and jenkins-infra
    • www.jenkins.io
      • jenkins-infra/jenkins.io
      • jenkinsciinfra/jenkins.io
  • => we want an exhaustive list of the images Ashutosh will produce jenkins/tutorial-controller- for DockerHub images
  • Source code: monorepo or multiple repositories?
    • Today, for Gscoc scope, we have 1 repo for all images
  • Lifecycle of releases
    • What is triggering a new image version?
      • A change to the code is merged
      • A new dependency is available
    • Then what would be the trigger to update jenkins.io tutorial?
      • Automatic dependency update detected (dependabot or updatecli)
  • CI?
    • ci.jenkins.io => Jenkinsfile, + action from jenkins-infra team to track the job for build
  • CD?
    • infra.ci.jenkins.io over trusted.ci?
      • infra.ci proposal, easier to operate and less security concerncs about the image content
  • jenkins-infra/helpdesk issue to open to track these tasks

• Done:

  • [Expiration of the Digital Oceans PATs September 2023
  • (September 2023) Update VPN CRL
    • Renewed for 6 months as usual
    • Calendar added
    • Stéphane updated the doc
    • GPG keys renewed for Stéphane and Hervé, also removed Olivier Vernin’s (olblak) outdated keys
  • IntelliJ plugin builds fail on Windows because Gradle cannot cleanup
  • JDK21: change from nightly builds to weekly ea builds in the infra
    • Upgrade done on both packer-image AND kubernetes-management AND Puppet
    • Also, version tracked with updatecli: we expect automatic PR to keep JDK21 up to date for now
    • Release of the official LTS JDK21 soon, might need to remove the -ea_beta suffix everywhere

• Work In Progress:

  • Assess Artifactory bandwidth reduction options

    • No more Apache Maven Central mirrored!
    • Logs receieved from JFrog so we can track impact on bandwidth
    • We expected positive impact after repopulating the cache
    • maven-hpi-plugin ITs are broken and Damien needs help to solve it

  • Proposal for application in publick8s to migrate to arm64

    • Fixed pipeline library: ATH image is build again with success and we keep arm64
    • List added in the issue, not absolutely exhaustive but a good start
    • updatecli must be aware of the arm64 images (and check both Intel and ARM)
    • We have to switch images with an sha (that we build) to the semver automatic deployment (instead of latest or manual or weekly updates)
    • Another improvement: speed up the Docker image library to create/push tags at the same time for both GH and Docker (instead of running additional build)
      • Issue to create by @smerle , the “why” is to decrease build costs from infra.ci

  • [publick8s] Ensure high availability of replicated services

    • PR for wiki to start with anti-affinity (in arm64 because more probable to happen as less nodes than x86_64)
    • Next step: we need a list of task to apply this

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • Digital Ocean primary mirror is being built
      • Worst case billing: 700$ per month (10x less than current AWS!)
    • Cloudlfare answered and is evaluating our sponsorship (for US and Asia mirrors)
    • Oracle for 1 year on US as secondary fallback
    • Deploy update center JSON index on all the mirrors

  • A few SSL certificates expire in 25 days or less

    • Problem is our AKS cert-manager ingress: details in issue, gotta work on it next milestone

  • Unexpected long delay uploading BOM artifact s3 bucket

    • New ci.jenkins.io pipeline behavior flag setup

  • nexus-jenkins-plugin bundles proprietary dependency

    • No action: removing from milestone

  • Remove account request field from Jira login page

  • LF status page redirect may be cached for too long

  • Matomo github/docker repos

    • New subnet is created for mysql
    • Draft for the terraform manager MySQL instance: need to discuss and validate some specs
    • Started to clean up Docker image, but no more work (yet)
      • arm64 soon?

  • [INFRA-3154] Certain emails from Jenkins mailing lists are failing SPF

    • not worked but @smerle might have some time to spend on

  • jenkins-infra/packer-images: Cleaning up playwright artifacts (fix(provision): clean playwright artifacts during sanity check by hainenber · Pull Request #801 · jenkins-infra/packer-images · GitHub)

    • @smerle might check and add additional context
    • Recommendation is to only cleanup artefacts after the sanity check

  • jenkins/packaging: Redirecting pkg.jenkins.io to install doc page (Redirect pkg.jenkins.io to install doc page instead of brief summary by hainenber · Pull Request #431 · jenkinsci/packaging · GitHub)

    • Non-infra topic

• ToDo (next milestone) (infra-team-sync-2023-09-19 Milestone · GitHub)