Infrastructure Team Meeting - June 28, 2022

[dduportal (Damien Duportal)]

2022 06 28 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly 2.357 is done

• Release checklist is nearly complete
• All good

Notes

• Done (infra-team-sync-2022-06-28 Milestone · GitHub)

  • Azure Service Principal expired for the AKS cluster temp-privatek8s (infra.ci)
  • Update Center stopped updating: job failing on trusted.ci.jenkins.io with error java.io.FileNotFoundException (File name too long)
  • Migrate doc from wiki to github for nutanix-calm plugin
  • Update Center is not updated (build failing on trusted.ci)
  • Jenkins 2.346.1 still warns about Jenkins Security Advisory 2022-06-22
  • Upgrade CIs to 2.346.1
  • Build failure javadoc cannot reproduce locally
  • Migrate from workflow-cps-global-lib to pipeline-groovy-lib
    • @lemeurherve checks for cert.ci with the security team
  • weekly.ci’s default view description diverged from the defined one
  • [ci.jenkins.io] Digital Ocean sponsorship runs out of credits: stopping the cluster

• Work in Progress (infra-team-sync-2022-06-28 Milestone · GitHub)

  • Cannot apt upgrade Jenkins from 2.348 using pkg.jenkins-ci.org
    • New task which involves adding HTTP redirection from old domain (pkg.jenkins-ci.org) to the new domain (pkg.jenkins.io) with enforced HTTPS to avoid such issues in the future, as a courtesy to people running instances since years with the “legacy” domains.
  • Claim that updates.jenkins.io certificate is sometimes expiring
    • Removing from milestone: that should be solved by [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  • Consider removing embeddable-build-status plugin
    • Almost there: need to notify developers on the mailing list
    • @lemeurherve mentions that Joseph and Tim have tooling and ideas about batch-opening pull requests for end users to propose an alternative and/or remove images that would be broken by this change
  • enable Development integration in JIra
    • Damien todo (need to check with jenkinsci admins for oauth creation :'()
  • [JENKINS-49707] Evaluate retry conditions to improve the stability of the builds
    • Damien todo: check with Jesse to test the plugins on ci.jenkins.io
  • Replace s390x Ubuntu 18.04 agent with s390x Ubuntu 20.04 agent
    • Ping to @en3hD3iMRx6_6IXLNY0Rag about the status of this one. s390x agent seems to be unreachable from ci.jenkins.io, which means “work in progress”
  • [ci.jenkins.io] Provides powershell on the Windows Docker images used on ACI
    • Changed target from “ACI” to “all agents”
    • WiP by @lemeurherve on VM templates, next steps: windows containers
  • DockerHub rate limiting
    • Defined VM agents on ci.jenkins to use public IP and stop logging in for CI
    • Downside: hitting the hard limit of public IP on Azure VMs
    • Damien to ping Docker
    • Damien to update VM agents configurations to NOT allocate a public IP on highmeme and ARM64 VMs (as Intel Linux/Windows are used to build Docker images)
  • Upgrade to Kubernetes 1.22
    • EKS + DOSK DONE \o/
    • Wip: AKS upgrades thursday 30, 07:00 UTC by @smerle and @lemeurherve
    • Prepare issue for kube 1.23 with learnings from this one
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • @smerle Did heavy lifting (terraform project init, retrieve networks, create data volume and backup policies, prepare shape and instance)
    • @dduportal took over to help on the IAM/policies part to allow Stephane focusing on Kubernetes 1.22 upgrade
    • WIP: Handing over back to Stephane once IAM/policies is finished

• New/Importants infra-team-sync-next

  • CI setup for jenkins-infra/plugin-health-scoring
    • @dduportal takes it, need to give documentation, tip and answer to Adrien. No need for more at first glance, should be quick
  • Require Java 11 infrastructure thread
    • Already worked on this (@basil and @timja did the heavy lifting). Adding to milestone to materializ the “WiP” + current weekly without JDK8 + to track JDK updates
    • Upcoming issue to start trying running infra.ci (and weekly.ci ?) with JDK17
  • [terraform:AWS] manage EKS modules as code
    • Caught by @smerle during EKS 1.22 upgrade: could improve the management of EKS modules. Not priority but nice to have
  • AKS: add cluster privatek8s
    • Next heavy task
  • [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
    • Next heavy task

• ToDo (next milestone) (infra-team-sync-2022-07-05 Milestone · GitHub)