Infrastructure Team Meeting - August 08, 2022

[dduportal (Damien Duportal)]

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @timja (Tim Jacomb)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly 2.363
   • Complete, checklist complete, Docker images confirmed built
   • Revision to the changelog is submitted
   • Git client plugin security fix for host key verification broke the packaging step
2. Costs
   • July expenses high on all platforms in July
     • May have missed autoscaling on kubernetes, manually scaled to complete a release, forgot to autoscale
     • Had machines running but idle
     • Fixed last Friday, reduced Azure cost estimate
   • AWS expenses at $12k per month
     • Needs more research to understand costs
   • DigitalOcean credits are low
     • Email notice to the DigitalOcean sponsors to increase
3. Salad Cloud
   • Seeking Advice: We have too much compute, not enough jobs!
4. Launchable
   • Kohsuke Kawaguchi would like to allow the Jenkins project to use Launchable
   • Would likely extend buildPlugin() to send test duration and result info to Launchable service

Upcoming Calendar

• Next Weekly: 2.364 - 16th of August 2022
• Next LTS: Tomorrow - 10th of August 2022 (infra stable tomorrow)
  • Alex Brandes is release lead
• Next Security Release: N.A.
• Next major event: N.A.

Notes

• Done:

  • [minor] infra.ci logs are mentioning an expired datadog API key
  • Some unexpected error occurred when executing shell script
  • Certificate Authority Authorization to secure certificate delivery
  • Jenkins 2.361 built with JDK 11.0.14 instead of 11.0.15 or 11.0.16
  • Creating a node fails on Windows when jenkins has not been installed on the C: drive
  • New Gradle Plugin Group ID requiring proof of jenkins.io domain ownership
  • Error when downloading plugins

• Work In Progress:

  • Puppet Upgrade Campaign to latest 6.x
    • We are using Puppet Enterprise, with the 10 node free license, not the OSS
    • Delayed to Thursday or Friday
  • Latest Trilead API plugin version 1.71.v9e7860a_67a_df fails when using JDK8 for agent process
    • Agent with JDK8: failing.
    • Fixed on ci.j and trusted.j.
    • But infra team has to ensure JDK11 is used explicitly for all agent processes
  • Consider no longer using http_request plugin
    • No infra action required. Let’s wait for update_center to be updated
  • Email, SPF, jenkins.io for Jenkins CERT bot
    • Delaying to Stephane return ~ middle-august
  • [INFRA-2697] Unfork repos
    • Managed by Alex and Tim. No more action expected from the infra-team.
  • migrating Pipeline jenkinsio-cn from trusted to infra.ci
    • Blocked by Publish `pipeline-steps-doc-generator` and `backend-extension-indexer` artifacts to some kind of storage · Issue #3087 · jenkins-infra/helpdesk · GitHub
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Blocked [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
  • ci.jenkins.io permissions for contributor
    • Daniel shared a short-term technique to allow testing Jenkinsfile changing in the issue
    • Josep is a contributor whose scope of action is broad. Let’s ask him to ask for more permissions for ci.jenkins.io on the mailing list (it’s not strictly necessary but sounds practical and useful)
  • Alert fatigue: Datadog + Pagerduty are too verbose
    • Herve and Stephane disabled the “false-positive” monitor.
    • Long running task now: removing the datadog custom image in favor of the helm chart (for custom checks) + migrating in full terraform the “weird response” time message with synthetics
  • Access to npm namespace
    • Damien to check this before end of week (hand over with holidays)
  • Replace s390x Ubuntu 18.04 agent with s390x Ubuntu 20.04 agent
    • @en3hD3iMRx6_6IXLNY0Rag to share SSH access to Damien
  • Want containerized Java 17 Windows agents
    • Half of the work done. Damien to handover from Herve to finish the packer windows container work

• New/Importants infra-team-sync-next

  • SECURITY-1468’s solution impacts ATH builds
    • Damien to comment there (and fix) tomorrow
  • Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage raised by Tim
    • Added to next milestone, blocks migration of *.jenkins.io to infra.ci (from trusted)
  • https://twitter.com/jenkins_release is many weeks behind
    • Looks like that we need help from Tyler to get access to the bot
  • [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
    • Blocks the updates.jenkins.io migration to Oracle: added to next milestone
    • After the LTS, ideally before next weekly.
  • Weekly release build does not resume
    • Should be easy: adding a retry in the pipeline. Next milestone, after the LTS release, before the next weekly.
  • Valid ssl certificate for trusted.ci.jenkins.io
    • Next milestone, should be easy. Extendable to cert.ci, but NOT to ci.jenkins.io.
  • Publish acceptance-test-harness docker image on release
    • Next milestone, should be easy.
  • [ci.jenkins.io] collect datadog metrics for ephemeral VMs
    • Next milestone: adding datadog agent to packer template. Eventually enable for VM agents.
  • [infra.ci.jenkins.io] Start using JDK17
    • Mentioned for info. Requires agent/packer/JDK17 availability to avoid weird issues (if any).
  • [ci.jenkins.io][Infra-as-code] Define Job Configuration as code #3071
    • For info, for later. Required for migrating ci.jenkins.io to Kubernetes.
  • [ci.jenkins.io][Infra-as-code] Define Core and plugins as code in a custom built Docker Image
    • For info, for later. Required for migrating ci.jenkins.io to Kubernetes.
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
    • Next milestone: install a set of instances in AKS, EKS and DOKS to start testing it on ci.jenkins.io. Goal: decrease bandwidth consumption from repo.jenkins-ci.org

• ToDo (next milestone) (infra-team-sync-2022-08-16 Milestone · GitHub)

[SaladBob]

Hi! it’s Bob from Salad here. Apologies for missing this, I wasn’t sure of the logistics and just got an email notification. Would love to join next week (please let me know how)…and I see the cost of AWS is an agenda item

[NotMyFault (Alexander Brandes)]

Are there (public) stats of AWS costs for comparision?

Tim and I cleared the backlog of dependency updates of the ATH earlier in July, which runs on highmem nodes only, plus we did a bunch of core PR testing, I’m wondering if there’s a coincidence.

[dduportal (Damien Duportal)]

Hi Bob, no problem. Feel free to join us today: the meeting is referenced in this page: Events (all of our meeting are public).

[dduportal (Damien Duportal)]

That could be linked to the overall cost increase (at least make sense).

Spending credits is not a bad things: in this case it means people worked on the topic.

But from infrastructure prospective, there are still 2 issues:

• A lot of failing agents triggered unwanted costs (by autoscaling worker nodes while pod where failing). Infra need to improve on this topic.
• Infra team forgot to disable manual scaling on Azure, which also costed few bucks.