Infrastructure Team Meeting - May 09, 2023

[dduportal (Damien Duportal)]

2023 05 02 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• Saarthak Maini

Announcements

1. Weekly: 2.404 is out (release, package and Docker image)
   • Release checklist to be finished (changelog, doc, etc.)

Upcoming Calendar

• Next Weekly: 2.405, 16 of May 2023
• Next LTS: 2.346.1, 22 June 2023
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event: N.A.

Notes

• Done:

  • Jacked plugin is not visible on https://plugins.jenkins.io/
  • Unable to create account
  • Renew update center certificate (crawler and update-center)
  • Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.387.3
  • Update center is missing Jenkins 2.403 weekly release
  • Don’t send PagerDuty notifications on Datadog warning notices

• Closed as not planned:

  • Debian packages starting with 2.387.2 are no longer published under https://pkg.jenkins.io/debian-stable/binary/
  • forgot username and password
  • Missing update center for 2.387.2

• Work In Progress:

  • Increase disk space for systempool on privatek8s
    • Changing this resource require destroy/recreate of the cluster
    • Let’s start moving “app” workload to decrease disk usage
    • We also upgraded linux pool 50 → 64 Gb
  • Jenkins job is failing to load the local jar from lib folder.
    • 2 PRs to exclude from ACP (puppet and kube) should solve the issue
  • Migrate trusted.ci.jenkins.io from AWS to Azure
    • 3 (empty) VMs in the same networks (bounce, controller and permanent) / subnet
    • Next: work on security groups, and then start preparing the migration data
    • Puppet works perfectly on Ubuntu 22.04 for the jenkins:controller: go go go to add these VMs once security groups are ok (and before migration
  • Can’t access applitools account
    • Should be ok, gotta check one last time
  • Make “Environment” and “Description” fields mandatory for “Bug” type issues
    • Asked Mark for help
    • Looks like there might be objection for the “Environment” (technical reasons) in the mailing list
  • Failed to Deploy Artifacts:… 401 Unauthorized when releasing plugin
  • password reset
  • Past Release sites are taking long time to load
    • Back to backlog as they are no emergency nor blocker now
  • migrate google analytics to v4
    • Back to backlogs until Olivier and Gavin answers backs
  • [ci.jenkins.io] Use a new VM instance type
    • Work in progress, size found, VM ready to be created, need PR and review
  • Migrate applications from systempool to linuxpool on privatek8s
    • Gotta work on this one (delayed by core releases and long week end)
  • cant create account
  • Add Launchable to agents
    • Python + launchable installed on the windows ACI’s nanoserver images, but has errors on runtime and does not work with launchable alone
    • Working around missing DLLs, trying to get them from windows server core but looks like a rabbit hole
    • Let’s check with Basil with the current state if it works, otherwise we’ll have to use windows server core as base image
  • Artifact caching proxy is unreliable
    • With the new inbound azure VM feature, we should be able to migrate ci.jenkins.io ephemeral agents to the ACP network
  • AKS: add cluster publick8s
    • Need to plan migrations (pick a service and move it, and iterate)
  • Temporary name resolution failures on plugin BOM builds
    • Need more investigation on the Core DNS embeeded in the EKS clusters
    • We removed the datadog agent custom probes on cik8s, need to the same eks-public
  • Cleanup and import unmanaged Datadog monitorings in terraform

• New items:

  • Azure ARM64 VMs
    • Work with success for infra.ci and packer
    • Next step for infra.ci: remove AWS arm64 template
    • Next step for ci.jenkins: do the same (AWS → Azure for arm64)
    • Last mile: cleanup packer-image to remove any AWS code
  • Ubuntu 22.04
    • AKS node pools

• ToDo (next milestone) (infra-team-sync-2023-05-16 Milestone · GitHub)