Infrastructure Team Meeting - May 03, 2022

[dduportal (Damien Duportal)]

2022 05 03 Jenkins Infra Meeting

Attendees

• @smerle33 (Stéphane Merle)
• @basil (Basil Crow)
• @dduportal (Damien Duportal)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly: 2.346 failed because of an expired credential on Azure. Credential rotated, but release need to be re-triggered.

Notes

• Done (infra-team-sync-2022-05-03 Milestone · GitHub)

  • DockerHub credentials
    • Pipeline library updated for new Docker credentials (pull/push separated)
    • Credentials updated
    • DockerHub accounts / org. documented
    • Docker Inc. accepted our request for OSS program: waiting for them (Apply to Docker Open Source Program · Issue #2842 · jenkins-infra/helpdesk · GitHub is WiP: let’s remove from milestones?)
  • [repo.jenkinsci.org] JFrog’s Artifactory WebUI is unavailable artifactory jfrog monitoring
  • JDK 11/17 Upgrade campaigns (17 tracked in JDK17 17.0.3+7 upgrade campaign, no tracking for 11 as it was automatic)
  • Kubernetes 1.21 Upgrade campaign
    • AKS clusters
  • Missing ARM64 / s390X Docker images for Jenkins Controller
  • [SSO for crowdin.jenkins.io](JDK17 17.0.3+7 upgrade campaign) =>) => won’t do as discussed last week

• Work in Progress (infra-team-sync-2022-05-03 Milestone · GitHub)

  • ci.jenkins.io outages:
    • 10 days ago: ci.jenkins.io outage (BOM build wave)
      • Post Mortem to be run, outcome is a set of tasks
    • Last week end (01/02 may) ci.jenkins.io slow, bunch of BOm build (dependabot)
    • Outage - ci.jenkins.io unresponsive due to CPU saturation · Issue #2908 · jenkins-infra/helpdesk · GitHub
      • Flamegraph taken thanks to @bcrow and @markewaite
        • Worth opening a runbook for “how to make flamegraphs”
      • Time spent on git clone of shared lib + compilation of shared library
      • Might be related: feat(ci.jenkins.io,trusted.ci.jenkins.io) ensure that pipeline shared library are not cached when on a Pull Request branch by dduportal · Pull Request #2142 · jenkins-infra/jenkins-infra · GitHub
        • Worth opening an issue with reproduction case to help maintainers
    • Top priority: allow Basic in
      • SSH access
      • Runbook access
      • Admin of Jenkins ci.jenkins.io
      • VPN account
  • Migrate rating.jenkins.io to AKS
    • Planned for the 03rd of May (morning EU)
    • Still gotta cleaned up the AWS resources (VM, data disk, database instance, etc.)
  • Build our own container agent Windows images on infra.ci
    • WiP on the pipeline library
    • WiP on a packer-image PR to install CLI tools for this library in windows VMs
  • infra-report/RPU
    • New issue on RPU, with low impact: RPU builds failing on trusted.ci for the `Archive` step due to missing `az` · Issue #2520 · jenkins-infra/repository-permissions-updater · GitHub
    • WiP on a packer-image PR to install AZ cli
  • Mirror in Singapore mirrors.jenkins.io
    • No work done, to be treated on next milestone
  • Replace Blue Ocean in default display URL (or remove the Blue Ocean plugins)
    • Asked end user vote on the mailing list
    • Stephane started to prepare an eventual PR to add the parameter for ci.j
  • Sunset mirrors.jenkins.io
    • No work done, to be treated on next milestone
  • [Apply to Docker Open Source Program](Apply to Docker Open Source Program dockerhub)
    • Waiting for Docker Inc. (they accepte but need to onboard our accounts in their new OSS program)

• New/Importants infra-team-sync-next

  • Datadog: deprecate @oncall handles
  • Datadog: monitor repo.jenkins-ci.org WebUI
  • Realign repo.jenkins-ci.org mission artifactory
  • Migrate updates.ci.jenkins.io to another Cloud
  • Proposal from Herve (not tracked yet): "Split Azure Terraform into 2 projects: “azure-net” and “azure”
    • Goal 1: Lower risk of breaking infra. foundation
    • Goal 2: increase our ability to iterate with trust on azure

• ToDo (next milestone) (infra-team-sync-2022-05-10 Milestone · GitHub)

  • Top priority: allow Basil in
    • SSH access
    • Runbook access
    • Admin of Jenkins ci.jenkins.io
    • VPN account
    • EKS cluster
  • Side tasks for Basil (not priority but to be done)
    • Report pipeline-library caching bug w/ reproduction
    • Runbook flamegraph