Infrastructure Team Meeting - November 08, 2022

[dduportal (Damien Duportal)]

2022 11 08 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)

Announcements

1. Weekly:

• Base release is OK
• Container image now available
• Rest of checklist item later today

2. 11 of November is banking day in part of the EU

Upcoming Calendar

• Next Weekly: 2022-11-15 (maybe?)
• Next LTS: 2022-11-30
• Next Security Release: 2022-11-15
• Next major event: FOSDEM (3-4 February 2023 in Brussels)

Notes

• Done:

  • Sign Up
  • Login
  • Missing datadog metrics for the prodpublick8s AKS cluster
  • [From Platform SIG] Remove PPC64 mentions on ci.jenkins.io
  • Artifact downloads failed on agent using repo cache
  • Access to npm namespace
  • Archive a few Jira components of NotMyFault’s plugins
  • Windows ACI 11 agent broken: no git found
  • [INFRA-2939] Request permission elevation on oracle-cloud-infrastructure-compute-plugin-developers
  • Archive TobiX’s Jira plugin components
  • https://twitter.com/jenkins_release is many weeks behind
  • ci.jenkins.io does not have enough GH permissions to check trust on jenkins-infra/jenkins-codeql
  • Hosting team onboarding
  • Upgrade to Kubernetes 1.23
  • Not receiving reset account password email
  • Crowdin localization for lockable-resources
  • update center json returning 404
  • password

• Work In Progress:

  • https://ci.jenkins.io/job/Infra/job/stories/ is not handling PRs
    • Missing permissions
    • New Job location on ci.jenkins.io
    • WiP: merge pipelines + packer image adding missing tools
    • Damien and Stephane
  • Login Issue
    • TBD
  • forget password
    • TBD
  • Windows agents on ci.jenkins.io disconnect prematurely
    • No incident
    • But label windows to be remove and/or acceptance test to be updated
  • Node.js missing on agent
    • Issue ASDF + NodeJS and the env.
    • Should we use official NodeJS?
    • Alternatives: NVM
    • => install node from official installer, first on docker-builder, then on packer images
  • Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage
    • Done, but need to enable CI for public visibility (same as stories)
    • Damien and Stephane
  • Artifact caching proxy unable to deliver git-changelist-maven-extension
    • 2 issues:
      • one is from maven central (and not ACP)
      • second comes from the azure’s ACP (HTTP/504). Checked in datadog: sounds like network issues on prodpublick8s (IP overlapping).
    • Let’s prioritize the privatek8S + publick8s fixups.
  • Add an help desk template for account recovery issues
    • Herve will open a draft open for feedbacks
  • Backlog of already built items in queue on ci.jenkins.io
    • Waiting for the 16th
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
    • 3 ACPs are running
    • We get datadog metrics and logs for all of them
    • Herve watching it, but priority on “Azure IP overlap” solving
    • Improvement on the pipeline-library:
      • Define “available” ACPs as env vars (feature flag)
      • Run an healthcheck before a build, to fallback to JFrog if current ACP is down
      • (and some minor fixes ;))
  • Migrate Keycloak database to Azure
    • New database created in Azure (on the current “public” postgres instance)
    • WiP: initial backup/restore
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
    • Migration to be defined, old instance to be kept one month
    • Bandwidth decrease: Mark and Damien to write and hack to find a proper path
    • Work on HA LDAP (at least read-only replica) to be started
  • AKS: add cluster privatek8s
    • Create a new cluster with terraform using a managed identity instead of a service principal
    • Migrate services from temp-privatek8s to this new cluster
    • Migrate services from prodpublick8s to this new private cluster
      • infra, release
      • monitoring
      • bots (only need egress)
  • (proposal) Add external-dns to the Kubernetes cluster with an ingress (and a loadbalancer)
    • Go back to the backlog!

• New/Important:

  • Keycloak performance horrific when looking up / modifying users
  • AKS cluster prodpublick8s: migrate persistent volumes from in-tree to CSI provisionners
  • Ubuntu 22.04 upgrade campaign
  • Valid ssl certificate for trusted.ci.jenkins.io
  • implement new test on check availability for agents
  • Windows agents are soooooooooo slooooooooooooooooooow
  • Add observability for the build agents
  • Update LTS issue filter search query
  • [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
  • Remote repository for repo.karuslabs.com
  • jenkinsci account verification on Twitter
  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
  • Want containerized Java 17 Windows agents
  • migrating Pipeline jenkinsio-cn from trusted to infra.ci
  • “Latest” links for some plugins on archives.jenkins.io are outdated
  • Downloads /latest directory out of date
  • Migrate Blue Ocean remaining jobs from ci.blueocean.io to the OSS Infra
  • Alert fatigue: Datadog + Pagerduty are too verbose
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  • [Azure Terraform] Import existing resources
  • Use cases and repositories for github-comment-ops on jenkins-infra org
  • [infra.ci.jenkins.io] Start using JDK17
  • enable Development integration in JIra
  • [ci.jenkins.io][Infra-as-code] Define Core and plugins as code in a custom built Docker Image
  • [ci.jenkins.io][Infra-as-code] Define Job Configuration as code
  • Puppet Upgrade Campaign to latest 7.x
  • [INFRA-3080] Migrate Windows Server from 2019 to 2022
  • temp-privatek8s cluster backup
  • [terraform:AWS] manage EKS modules as code
  • AKS: add cluster privatek8s
  • [INFRA-3135] Terraform 2021 Winter (/ 2022 Summer )
  • [INFRA-3137] Terraform: Import unmanaged Oracle Cloud resources
  • [Documentation] add a public page with the “add a jenkins mirror” procedure
  • GC AWS Old Images (from packer)
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • Separate terraform backends & repositories: “Azure Net” and “Azure”
  • [INFRA-3125] Migrate jenkins-infra repositories from branch “master” to “main”