Infrastructure Team Meeting - September 13, 2022

[dduportal (Damien Duportal)]

2022 09 13 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly build
   • 2.368 released, changelog visible, container ready

Upcoming Calendar

• Next Weekly: Next week, as usual
• Next LTS: ~ October 5th (Kris Stern is release lead)
• Next Security Release: Wednesday 21st of September
• Next major event:
  • DevOps World in 2 weeks= jenkins infra meeting cancelled (27th of September) - @damien to remove the invitation from calendar

Notes

• Done:

  • Build failing for packer-images
  • Approve jenkins.io domain on the GitHub organizations
  • [Jira] Remove jglick (Jesse Glick) as default assignee for pipeline-groovy-lib-plugin
  • Archive custom-distribution-service
  • Weekly release build does not resume
  • Archive extras-executable-war
  • [ci.jenkins.io] Outage on the cik8s Kubernetes cluster: no more Linux builds for plugins and BOM
  • Upgrade CI’s to 2.361.1
  • Delete my account in accounts.jenkins.io
    • Updated challenges (this one + GH fork) in https://github.com/jenkins-infra/runbooks/tree/main/accounts#delete-a-user: https://github.com/jenkins-infra/runbooks/pull/64

• Work In Progress:

  • “Latest” links for some plugins on archives.jenkins.io are outdated
  • Publish acceptance-test-harness docker image on release
  • Hosting Plugin Health Scoring application on infra
    • CI part for building Docker workflow (stash/unstash) Need tests and good to merge
    • Adrien working on the helm-chart
    • DNS plugins.health.jenkins.io (currently redirecting to their gh page for now, until service is running in Kubernetes)
    • Upcoming need for a database (see Stephane for Azure managed Postgresql)
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
    • Azure proxy instance ok for health check and storage
    • Digital Ocean incoming
    • Next steps: test it (jenkins-infra-test-plugin + Herve to search a “big plugin” in term of deps.)
  • Damien is admin of JFrog now, KB with Daniel
    • Todo @dduportal runbook with the “rules”
    • Todo @dduportal open an issue about a virtual repo in JFrog which has no upstream (e.g. only cached file, that can be lost)
  • Missing datadog metrics for AKS clusters
  • https://twitter.com/jenkins_release is many weeks behind
  • [ci.jenkins.io] collect datadog metrics for ephemeral VMs
    • Azure is ok
    • EC2 working since yesterday
    • Last testing phase and done
  • Want containerized Java 17 Windows agents
    • All-in-one container (linux): almost there. Currently battle testing on jenkins-infra-test plugin.
    • Windows Container: provisioning failing, but almost there
  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
    • GH org to be implemented on helm chart
    • PoC on kubernetes-management
  • Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage
  • [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
  • Incoming work by @lemeurherve to have an IRC bot “archive” command
    • feat: add `archive repo` command by lemeurherve · Pull Request #111 · jenkins-infra/ircbot · GitHub

• New/Important:

  • Access to npm namespace
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
  • Downloads /latest directory out of date
  • Migrate Blue Ocean remaining jobs from ci.blueocean.io to the OSS Infra
  • Valid ssl certificate for trusted.ci.jenkins.io
  • Alert fatigue: Datadog + Pagerduty are too verbose
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  • migrating Pipeline jenkinsio-cn from trusted to infra.ci
  • [Azure Terraform] Import existing resources
  • Use cases and repositories for github-comment-ops on jenkins-infra org
  • [infra.ci.jenkins.io] Start using JDK17
  • enable Development integration in JIra
  • [ci.jenkins.io][Infra-as-code] Define Core and plugins as code in a custom built Docker Image
  • [ci.jenkins.io][Infra-as-code] Define Job Configuration as code
  • Puppet Upgrade Campaign to latest 7.x
  • [INFRA-3080] Migrate Windows Server from 2019 to 2022
  • Upgrade to Kubernetes 1.23
  • Ubuntu 22.04 upgrade campaign
  • temp-privatek8s cluster backup
  • [terraform:AWS] manage EKS modules as code
  • AKS: add cluster privatek8s
  • [INFRA-3135] Terraform 2021 Winter (/ 2022 Summer )
  • Add observability for the build agents
  • [INFRA-3137] Terraform: Import unmanaged Oracle Cloud resources
  • [Documentation] add a public page with the “add a jenkins mirror” procedure
  • Keycloak performance horrific when looking up / modifying users
  • GC AWS Old Images (from packer)
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • Separate terraform backends & repositories: “Azure Net” and “Azure”
  • [INFRA-3125] Migrate jenkins-infra repositories from branch “master” to “main”

• ToDo (next milestone) (infra-team-sync-2022-09-20 Milestone · GitHub)