Infrastructure Team Meeting - August 23, 2022

[dduportal (Damien Duportal)]

2022 08 23 Jenkins Infra Meeting

Attendees

• Bob Miles from salad.com
• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @MarkEWaite (Mark Waite)

Salad.com (guest: Bob Miles)

• https://salad.com/. Ref. Seeking Advice: We have too much compute, not enough jobs!
• Proposal: Jenkins project to use the project to get feedbacks for the workload
• Network of (personnal) computers running an agent process. Compute is distributed across this network of computers for different workloads: mining since 3 years, ability to run containerized workloads
• Average time of a task (given what JenkinsCI build are doing)? Incentive about “finishing” tasks but spectrum is broad
• Electric consumption: choice and measure delegated to the compute provider users, so no metrics
• Encryption: None, based on trust model (multi layers of trust indicators). Do not put sensitive dataset there.
• Workload isolation: Windows Host machines. Relies on WSL2 to tap in the GPU and Linux workload. Containers are run in the WSL isolation engine.

Announcements

• Damien in PTO from 27th until the 6th of september (included)
• Mark in PTO next Tuesday (25th)
• Security Advisory in progress, everyhting looks good
• Weekly Core release 2.365, checklist in progress, everything looks good

Upcoming Calendar

• Next Weekly: Next Tuesday
• Next LTS: 7th of September, dropping JDK8 for this LTS
• Next Security Release: today
• Next major event: N.A.

Notes

• Done:

  • UC doesn’t offer release of new plugin
  • JDK8 images only published for first patch release 2.346.1 of final release cycle
  • Migration of jenkins(se) from on prem to google cloud vmware engine
  • [accounts.jenkins.io] Fix instructions for spam unblock
  • ec2-plugin patch by @lemeurherve to specify custom java bin path
    • https://github.com/jenkinsci/ec2-plugin/pull/766
    • tested with an incremental build, deployed on infra.ci.jenkins.io
    • allow us to specify the agent JDK to use, for every OS (wasn’t possible for Windows before. Note: we weren’t concerned by this issue as we’re using SSH for our Windows agents, using the unix template)

• Work In Progress:

  • incorrect/missing maven settings file (for plugin builds?)
  • Enter too many times CAPCHA code
    • How does the blocking works?
    • Wait for security feedback
  • Publish acceptance-test-harness docker image on release
    • Almost there: we are almost there. Trying deploy with tag to be done before closing.
  • [ci.jenkins.io] collect datadog metrics for ephemeral VMs
    • Datadog agent enabling validated manually in Azure
    • Currently adding the setup on JCasc for controllers
    • Next step: EC2 VMS
  • Want containerized Java 17 Windows agents
    • Merged VM and Container templates into the same one. Linux to be tested (and deployed) if working. @lemeurherve
    • Windows container work in progress. Nitpicking on powershell.
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
    • Goal: limit bandwidth consumed from repo.jenkins (JFrog)
    • Re-introducing the same as years before, but 1 proxy per cloud region (to avoid cross-region bandwitdh)
    • 1st instance installed in Azure, tests in progress to get the correct Maven settings to use it (locate the correct proxy depending of the agent type. Using env. var from agent template)
    • Tested on the jenkins-infra-test-plugin (on real life ci.jenkins.io)
    • Next steps: Add user/password protection (in addition to IP whitelisting) and deploy the 2 other proxies and their configuration
  • https://twitter.com/jenkins_release is many weeks behind
    • Account recovered.
    • Next step: understand dlvr and check the actual automation to fix it
    • Note from Hervé: could be a good help to automate tweets for advocacy
    • Damien asks for help. @lemeurherve volunteers
  • Access to npm namespace
    • Waiting for the NPM support team. Removing from milestone until they answer.
    • There is a jenkins user with no plugins and no activity: trying to reclaim this account to conert it to an organization (and fix the request from Gavin)
  • Weekly release build does not resume
    • Fundamental question: do we want automatic retry? Is it worth it given the frequency.
    • Agent failure? Even this is questionnable for release.
  • [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
    • Might be slowed down with Damien PTO
    • Required puppet factorization

• New/Important:

  • Downloads /latest directory out of date
  • Valid ssl certificate for trusted.ci.jenkins.io
  • Alert fatigue: Datadog + Pagerduty are too verbose
  • Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  • migrating Pipeline jenkinsio-cn from trusted to infra.ci
  • [Azure Terraform] Import existing resources
  • Use cases and repositories for github-comment-ops on jenkins-infra org
  • [infra.ci.jenkins.io] Start using JDK17
  • enable Development integration in JIra
  • [ci.jenkins.io][Infra-as-code] Define Core and plugins as code in a custom built Docker Image
  • [ci.jenkins.io][Infra-as-code] Define Job Configuration as code
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
  • Puppet Upgrade Campaign to latest 7.x
  • [INFRA-3080] Migrate Windows Server from 2019 to 2022
  • Upgrade to Kubernetes 1.23
  • Ubuntu 22.04 upgrade campaign
  • temp-privatek8s cluster backup
  • [terraform:AWS] manage EKS modules as code
  • AKS: add cluster privatek8s
  • [INFRA-3135] Terraform 2021 Winter (/ 2022 Summer )
  • Add observability for the build agents
  • [INFRA-3137] Terraform: Import unmanaged Oracle Cloud resources
  • Migrate Blue Ocean remaining jobs from ci.blueocean.io to the OSS Infra
  • [Documentation] add a public page with the “add a jenkins mirror” procedure
  • Keycloak performance horrific when looking up / modifying users
  • GC AWS Old Images (from packer)
  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)
  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
  • Separate terraform backends & repositories: “Azure Net” and “Azure”
  • [INFRA-3125] Migrate jenkins-infra repositories from branch “master” to “main”

• ToDo (next milestone) (infra-team-sync-2022-08-30 Milestone · GitHub)