Infrastructure Team Meeting - March 07, 2023

[dduportal (Damien Duportal)]

2023 03 07 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: delayed to tomorrow as part of the 08 March 2023 security advisory
2. 08 March 2023 security advisory
3. From Platform and Documentation SIGs: we would want to put an additional effort on deprecating the “End Of Life” things such as CentOS-7, JDK8, Ubuntu 18.04. For VMs, containers, and also documentations.
   • A JEP will be written
4. Documentation SIG decided that the installation guide will be shifted from JDK11 to JDK17 (April 2023)
   • (Damien says: “Infra should lead this way by switching to JDK17”)
5. JDK updates in April (21th): JDK20 (but also JDK11/17). No date for JDK21 LTS line though

Upcoming Calendar

• Next Weekly: 14 of March (2.395)
• Next LTS: tomorrow (08 of March) 2.387.1
• Next Security Release as per jenkinsci-advisories: 08 March 2023 security advisory
• Next major events:
  • ScaleX at LA this week
  • Devoxx France April 12th to 14th

Notes

• Done:

  • Azure Service Principal expires the 08th of March for the AKS cluster prodpublick8s
  • Attempt to skip artifact caching proxy failed
  • Failed plugin builds due to missing Windows configuration file
  • 502 Bad Gateway error on Azure artifact caching proxy
    • Created Monitor the ACP disk usage · Issue #3432 · jenkins-infra/helpdesk · GitHub to work on the monitoring
  • Frequent PagerDuty Alerts Disk space is below 1GB free win-xxx-yyyyyy
  • Maven 3.9.0
  • Old email provider deleted
  • Remove Docker build for jenkinsci/plugin-compat-tester
  • Jenkins Core Release: disable weekly release for 07th of March 2023
  • Unable to login
  • updatecli mixing indentations
    • Updatecli 0.46.0 solved the issue
    • Can be closed (Stéphane to add a comment and close)

• Work In Progress:

  • Accounts related Issues:
    • We can’t access our plugin account
  • Non “directly actionnable” issues:
    • Migrate jan-molak/jenkins-build-monitor-plugin to jenkinsci GitHub organization
  • “Expired Credentials” issues:
    • EC2s are not available
    • trusted.ci doesn’t spawn new nodes
    • (2023-02-15) Azure Credential for cert.ci.jenkins.io is expired
    • ci.jenkins.io Azure credentials also expired (same as cert.ci: temp credential + candidate to WiM)
  • Jenkins Code Signing Certificate related issues:
    • Document the code signing certificate renewal process
    • Renew the signer certificate for jenkins
      • Mark to send a reminder to Digicert (+ adding jenkins-infra-team@googlegroups.com in Cc.)
  • Jfrog related issues:
    • (Re) Introduce an artifact caching proxy for ci.jenkins.io
      • pipeline-steps-generator + backend-extension-indexed is
      • Next repository (wip): jenkins-infra/javadoc
      • Impact of ACP: positive \o/ We’re no longer on the top 5 consumers
    • [INFRA-2754] Realign repo.jenkins-ci.org mission
      • LDAP-HA: WiP on Kubernetes
      • Data from Jfrog still being processed to demonstrate if we need to enable auth due to “maven repo1” cache or if it is related to “Jenkins activity” (e.g. a mis-configuration POM).
        • We should check that we do not have our public IP in this list
      • Main priority is focusing on the main consumer’s (which is not us anymore) and we need help of Jfrog as our attempts to contact these consumer(s) failed
    • Maven central packages not found on build agent
  • Others:
    • maven-17 label hanging on ci.jenkins.io
      • Gotta close and open a separated issue to split workloads
    • Update center job is failing
    • jobcopy-builder plugin 1.5.0 release fails to download from updates.jenkins.io
    • AKS: add cluster privatek8s
      • A few “bot” services (irc, rss2twitter and github-command-ops) to be migrated from prodpublic to privatek8s.
        • To be announced
        • Should require a service downtime for migration. No risk to do that.
    • Create an updatecli manifest to update the kubernetes quota depending of the ci config
    • Grant limited access to release.ci to some security team folks
    • Out of space on a ci.jenkins.io agent in bom build
    • Replace updatecli GitHub Action by a dedicated Jenkinsfile
      • WiP by Stephane. Almost there for adding the new Jenkins pipeline (both JobDSL and Jenkinsfile) \o/
      • Next steps: remove GHA
    • Check if we could replace blobxfer by an az-cli command
      • Back to backlog

• New issues to consider for next milestone:

  • Proving I am not spammer
  • Ubuntu 22.04 campaign
    • pkg.origin.jenkins.io
    • packer_images
  • Migrate static services to Azure
    • We have enough budget on Azure
    • trusted.ci (3 VMs)