Infrastructure Team Meeting - June 07, 2022

[dduportal (Damien Duportal)]

2022 06 07 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)

Announcements

1. Weekly 2.351: issues
   • Infra issues (Kubernetes ↔ Jenkins configuration)
   • docker/packaging ubuntu 22.04. createrepo → createrepo_c. Issues: Openssl + JDK changes.
   • Let’s continue to watch this release.

Notes

• Done (infra-team-sync-2022-06-07 Milestone · GitHub)

  • Nginx 1.22
    • Done everywhere, no need for ingress as it is still using 1.19
  • Build our own Windows Docker Images
    • Great job! A lot of pipeline-library and powershell tricky tricks
    • Deployed to ci.jenkins.io with no problem
    • Note: we broke infra.ci while trying to execute Windows pod agents (we added a node pool of Windows type, and kubernetes tried to reschedule infra.ci, which led to failure in diskchk for the data volume)
  • Update center certificate expired + crawling
    • Reminder: Only Olivier, KK and Oleg have the CA key.
    • Olivier generated a new cert and uploaded it to trusted.ci for us
    • Doc + runbook + calendar up to date
    • Jobs fixed in trusted.ci \o/
  • Use docker instead of img by default
    • Looks good! Faster builds/tests (but more VMs)
    • Let’s remove img now!
  • ci.jenkins.io: All highmem agents are stuck
    • Many thanks Alex for notifying us!
    • Cause was a new behavior in the azure-vm plugin. Rollbacked on the plugin by Tim (but new release not installed yet)
    • We changed the retention policy for azure VM to “use only once”: sounds like it mitigated the issue
  • Weekly 2.350 failed due to missing mb command
    • Consequence of mirrorbain sunsetting.
    • Fixed in the release scripts at https://github.com/jenkins-infra/mirror-scripts
    • More mirrorbrain cleanup is needed: [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain · Issue #2970 · jenkins-infra/helpdesk · GitHub
  • release.ci.jenkins.io is down
    • Kubernetes management PR merged which restarted the controller
  • IRC cloak for libera
    • Thanks @timja

• Work in Progress (infra-team-sync-2022-06-07 Milestone · GitHub)

  • Docker Rate Limit blocking jenkinsci/docker builds
    *
  • bootstrap terraform oracle project
    • Blocks the “update center in oracle” and “import oracle” resources tasks
    • States in Azure, we are able to manage user and API key with terraform as well. Wip in secrets to be stored in SOPS, then we init. the project
    • Main issue: due to the azuread SSO, it’s tricky to get an API key. Gotta need Mark help (as initial admin). Short term: we used a local technical user with MFA.
  • Digital Ocean sponsorship
    • Thanks for checking the costs @lemeurherve ! Sounds good
    • We have to take an appointment with them
  • Kubernetes 1.22
    • WiP: kubectl upgrade to 1.22: updatecli is OK ([updatecli] [kubectl] Update version to 1.22.x by jenkins-infra-updatecli[bot] · Pull Request #105 · jenkins-infra/docker-helmfile · GitHub)
    • Blocked by https://github.com/jenkins-infra/kubernetes-management/pull/2403
  • Remove img
    • PRs opened on pipeline-library and docker-builder
  • “update center in oracle”
    • Todo, blocked by bootstrap terraform oracle project
  • Auto notify people based on service routing rules
    • Not worked on it yet

• New/Importants infra-team-sync-next

  • docker-packaging update: impact on weekly. Issue to be written @dduportal - Weekly release 2.352/2.353/2.354 is failing · Issue #2978 · jenkins-infra/helpdesk · GitHub
  • infra.ci backups @lemeurherve - temp-privatek8s cluster backup · Issue #2976 · jenkins-infra/helpdesk · GitHub
  • https://issues.jenkins.io/browse/JENKINS-49707 for retrying builds when agents are trashed (spot instances, etc.)
  • Some pipeline libraries minor test fixes for @lemeurherve - https://github.com/jenkins-infra/pipeline-library/pull/400

• ToDo (next milestone) (infra-team-sync-2022-06-14 Milestone · GitHub)

[halkeye (Gavin Mogan)]

@dduportal

Only Olivier, KK and Oleg have the CA key.

This might be something worth putting in our 1password subscription?

[dduportal (Damien Duportal)]

As I understood @danielbeck and @olblak explanations, the reason to have a shot list of key owner is part of the threat model. Once you have access to this CA key, it cannot be revoked (since it is kinda the of the root certification), compared to access to the UC certificate key.
Putting it in a 1 password subscription feels like opening it to a wider range of people.

I wouldn’t mind backuping the key in a GPG encrypted format in a private repository that only the owner could decrypt so it would not be stored only on their machines.

Also, we might want to request the board to discuss this topic: should be add owners considering that the 3 of them are not expected to be day-to-day responsive.

It’s risky to give this ownership to the infra officer as it might change yearly. Maybe the security officer eventually along with 1-2 board members?