Infrastructure Team Meeting - May 02, 2023

[dduportal (Damien Duportal)]

2023 05 02 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• @MarkEWaite (Mark Waite)

Announcements

1. Weekly: 2.403 is out, packaging worked on the first try! and container is out.
   • Next release checklist elements to be done later (merge of the changelog)
2. Update Center certificate
   • Need to catch back plugin releases
   • Disable the check (at least for 1 week)
   • Need Olivier to send us the proper key
3. Azure maintenances
4. Software update campaigns:
   • Ubuntu 22.04 => before end of May 2023 except for packaging. Might be worth considering Canonical extended support if we are stuck (but would be temp.).
   • Kubernetes 1.25 => Let’s target June 2023 (EOL July 2023 for 1.24 on DO/Azure)
   • Puppet 7 => proposal to do it after Ubuntu 22 and before Kubernetes 1.25

Upcoming Calendar

• Next Weekly: 9 May 2023: 2.404
• Next LTS: tomorrow (3 May 2023): 2.387.3
  • Kris Stern is release lead and plan to start at 04:00am UTC
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • cdCon, 8/9 May in Vancouver, @en3hD3iMRx6_6IXLNY0Rag @halkeye and @notmyfault will be there :party:

Notes

• Done:

  • [ci.jenkins.io] separate container agent resources between bom and other builds
  • [Azure Billing] Excessive consumption on *-packer-builds resource groups
  • [Azure Billing] Excessive consumption on eastus-cijenkinsio resource group
  • Error Available disk space is below 19.996% free for puppet.jenkins.io
  • Grant commit permission
  • Search on plugins.jenkins.io crashes on particular search strings
  • Having validation issue on PR
  • Login issue on Artifactory
  • assets.ci.jenkins.io HTTPS certificate expires 22 May 2023
    • Issue to be open for monitoring from datadog (as we already do it in ci.jenkins.io)
  • Password reset email not coming through
    • Mailgun provide us visibility
  • Releases of cdevents plugin not published to update center
  • Switch mail provider from Sendrid to Mailgun for accounts.jenkins.io and admin.accounts.jenkins.io
    • Warning: check amount of emails sent per month to stay on the free plan. Reminder end of month => @lemeurherve to add on ine jenkins-infra-team
  • [ci.jenkins.io] outage on the AWS cik8s cluster

• Closed as not planned:

  • The upload to the Maven repository fails with “401 Unauthorized”
  • [DigitalOcean] Notice of SSH bruteforce activities

• Work In Progress:

  • can’t create account
  • Artifact caching proxy is unreliable
  • Migrate trusted.ci.jenkins.io from AWS to Azure
    • Almost done for 2nd VM (controller), private Ip setup to fine-tune
    • Next step: SG and permanent agent
  • Add Launchable to agents
    • WiP on the Windows Containers (others are OK).
    • Blockers around the “install launchable step despite python being installed”
    • Need pair session to unblock
    • Next step: pipeline-library update ready to roll (with @basil review/approval) including enabling for Windows builds
  • [ci.jenkins.io] Use a new VM instance type
  • Migrate applications from systempool to linuxpool on privatek8s
    • Let’s add node selector first (and taint/toleration later)
  • Increase disk space for systempool on privatek8s
    • Gotta try with ubuntu 22.04 and Green/blue deployment
    • coredns: warning!
  • Migrate google analytics to v4
    • Need help from Olivier Vernin
  • Past Release sites are taking long time to load
    • Postpone as we need to update ubuntu 22.04 (kernel upgrade with better SMB support)
  • Make “Environment” and “Description” fields mandatory for “Bug” type issues
    • @dduportal or @en3hD3iMRx6_6IXLNY0Rag
  • AKS: add cluster publick8s
    • @lemeurherve takes it

• New items/ Items from Backlog:

  • Spring 2023: Decrease AWS costs
  • Puppet Upgrade Campaign to latest 7.x
    • Wait for Ubunut 22.04
  • Feat(Infra.ci): add Azure ARM64 VMs
    • @smerle to select a job in infra.ci to test with arm64
  • Renew update center certificate (crawler and update-center)
  • Ubuntu 22.04 upgrade campaign
  • [pkg.origin.jenkins.io] puppet agent keeps updating the GPG
  • [ci.jenkins.io] Azure billing shows huge cloud cost due to outbound bandwidth
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  • [ci.jenkins.io] Use Websockets for agents
  • [INFRA-2754] Realign repo.jenkins-ci.org mission

• ToDo (next milestone) (infra-team-sync-2023-05-09 Milestone · GitHub)