Infrastructure Team Meeting - May 23, 2023

[dduportal (Damien Duportal)]

2023 05 23 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• Igor

Announcements

1. Weekly: slightly delayed, should be ok later today, no infrastructure action (except watching)

Upcoming Calendar

• Next Weekly: 2.407, 30 May 2023
• Next LTS: 2.401.1, 31 May 2023, Alexander Brandes (@NotMyFault) is the release lead
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event: N.A.

Notes

• Creating tag to trigger release of Jenkins Docker image: should we grant “Maintainers” permission on jenkinsci/docker to the Jenkins Release

  • Proposal to add Alex (to make the upcoming LTS release safe)
  • Proposal to automate the process in the future to avoid these permissions => issue to open, work to do by infra task

• Done:

  • DigitalOcean leftover disk to cleanup ?
  • [Azure Billing] Excessive consumption on eastus-cijenkinsio resource group
  • Add Launchable to agents

• Work In Progress:

  • Feat(Infra.ci): add Azure ARM64 VMs
    • Condition checking thet the Gallery are updated before launching updated
    • Closable ? One last mile: cleanup EC2 plugin on infra.ci and credentials
    • Good impact on the AWS billing
  • Upgrade to Kubernetes 1.25
    • This week: let’s start with DO, eventually AWS
    • Wait for finishing prodpublick8s → publick8s migration for AKS
  • AKS: add cluster publick8s
    • Preparation work
    • Tested Redis connection on the new cluster (preparing get.jenkins.io)
    • Started working on the PostgreSQL DB migration
    • Handover to @dduportal
    • Incrementals Published (stateless, easy to migrate) to do
    • Keycloak to test martch the database
  • [Azure Billing] Peak of usage cost for the prod-core-releases Resource Group
    • Wait for kube 1.25 and publick8s migration
  • Migrate trusted.ci.jenkins.io from AWS to Azure
    • Puppet is ok for 3 VMs
    • Fixups on security groups
    • Nexts steps:
      • Data migration from AWS => @dduportal with @smerle secondary
      • security groups to fine tune for permanent agent => @smerle with @dduportal secondary
  • [ci.jenkins.io] Use DigitalOcean VM agents instead of container agents
    • Initial build of a VM custom image built with packer (close to our setup) but not fully automated yet (PoC)
    • No Linux ARM64, No Windows in DO, but at least “normal” and “highmem” VMs
    • Defer in 2 weeks
  • Install and configure Datadog plugin on ci.jenkins.io
    • WiP to enable communication ci.j container ↔ host datadog agent
    • Need team sync
  • Cleanup and import unmanaged Datadog monitorings in terraform
    • 2 old monitor not applicable anymore => delete + create issue stating the “problem to solve”
    • All the other monitors are now managed by Terraform
  • [ci.jenkins.io] Use a new VM instance type
    • Waiting for Puppet + Security Groups tuning
    • Destroy the “temp” resources, and then wait for publick8s + trusted.ci tasks to be finished
    • Defer in 2 weeks (hopefully)
  • Artifact caching proxy is unreliable
    • Still issues on Azure, with the “overlapped” network (might or might not be the cause)
    • Next step: migrate VM agents of ci.jenkins.io to new network
      • Tested manually on ci.jenkins.io new inbound mode for VMs

• ToDo (next milestone) (infra-team-sync-2023-05-30 Milestone · GitHub)