Infrastructure Team Meeting - April 18, 2023

[dduportal (Damien Duportal)]

2023 04 18 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)

Announcements

1. Weekly:

• 2.401 release is ok
• Packaging failed (same error as last week: a replay will solve the problem. Azurefile POSIX permission bug.)
• Docker image to be built
• Incoming changelog (and last checklist item) later today

Upcoming Calendar

• Next Weekly: 2.402, April 25
• Next LTS: 2.387.3, May 3
  • RC tomorrow (April 19)
  • Backporting started
  • Kris Stern is release lead
• Next Security Release as per jenkinsci-advisories: None
• Next major event:
  • CdCon, May 8-9, Vancouver

Notes

• Done:

  • Update code-coverage plugin on ci.jenkins.io
  • @jenkins_release suspension notice
    • Account plan changed to avoid suspension. This new plan looks ok for our publication rate.
  • Enable renovate for jenkins-infra/stories
  • Move awestruct-ibeams into jenkins.io repository
  • [Puppet] Datadog agent does not collect Apache logs
  • Jobs with tasks using azure upload are failing
  • update_center doesn’t build because agent-1 is offline
  • Archive EOL js components
  • Add NotMyFault to Copy Editors team
    • Also to the Docker team on GH jenkins-infra
  • [ci.jenkins.io] Powershell on Windows 2019 VM agents is broken (powershell.exe : Install-NuGetClientBinaries : Exception...)
  • [monitoring] Improve datadog tagging for (puppet) VMs to ease operation during incident
  • Jira account locked
  • GC AWS Old Images (from packer)
  • issue
  • Document the code signing certificate renewal process
  • Renew SSL certificate for updates.jenkins-ci.org
  • CodeSonar plugin Jira component lead reassignment

• Work In Progress:

  • Renew update center certificate (crawler and update-center)

    • Asked Olivier to generate certificate for us
    • Asking the board if Damien may have the CA key (along with KK, Oleg and Olivier)

  • Email sending for accounts.jenkins.io:

    • Mailgun account OK
      • Risk with a “free account”: as per Olivier Vernin, it uses “shared IP” that could be blocked by some provider
    • Answers from K, Olivier Vernin and Tim Jacomb about sendgrid:
      • KK: Current sendgrid plan only allow 1 admin.
      • Tim: Azure provides allows spinning up sendgrid managed account. Pro: centralized billing and easy to manage.
      • Olivier: Consider different sending solution in the lens of cost (Azure’s sendgrid could be expansive)
    • 2 issues (currently) open, blocked by this:
      • cant create account
      • Password reset email not coming through

  • Azure costs:

    • [ci.jenkins.io] Use Artifact Manager to store archived artifacts (and stashes)
    • Feat(Infra.ci): add Azure ARM64 VMs
      • Success in generating a Linux ARM64 Ubuntu 22.04 with our packer setup on Azure Gallery
      • Blocker with the Packer’s Azure publication, surfaced by the ARM64. Work in progress.
    • Spot instance
      • For Azure VM agents of ci.jenkins.io (ATH or projects needing Docker)
      • For Packer VM builds

  • AWS costs - Spring 2023: Decrease AWS costs

    • Migrate trusted.ci.jenkins.io from AWS to Azure
      • Resources listing done
      • 2 PRS for Terraform Azure and Azure Net
      • Next step: pair review for network and stuff
    • [ci.jenkins.io] separate container agent resources between bom and other builds
    • Artifact caching proxy is unreliable
      • Bom: is the pct/sh in need of ACP?
      • ATH: see ci.jenkins.io network changes

  • Make “Environment” and “Description” fields mandatory for “Bug” type issues

  • [pkg.origin.jenkins.io] puppet agent keeps updating the GPG

  • [infra.ci.jenkins.io, ci.jenkins.io] ARM64 VM agent unavailable

  • [ci.jenkins.io] Define a default build discarder policy for all jobs

  • [datadog] Change the “Disk space is below 1GB free” monitor to a “80% disk usage”

  • Add Launchable to agents

    • PR ready, looks good to roll (then let’s release)

  • Ubuntu 22.04 upgrade campaign

    • docker-openvpn: PR opened, to be checked

• New topics:

  • Reduce amount of bom builds (less tests in PR, and group main branch builds)
    • Allow an “opt-in” in PR
    • Issue to open in helpdesk
  • Octobox installation request by Gavin Mogan (@halkeye)
  • [ci.jenkins.io] Performances
    • HDD → SSD (issue todo)
    • Terraform import
    • Remove JobConfigHistory plugin in favor of a daily disk snapshot
    • Move to the new public network (along with ci.jenkins.io Azure VM agents and ACI)
    • Shrink the “Azure (.*) Agent” template to optimize Azure API client pool
    • Enable websockets for agent

• ToDo (next milestone) (infra-team-sync-2023-04-25 Milestone · GitHub)