Infrastructure Team Meeting - April 09, 2024

[dduportal (Damien Duportal)]

2024 04 09 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @MarkEWaite (Mark Waite)

Announcements

1. Weekly (2.453) is out

   • Release went well
   • Packaging script was stuck on OSUOSL sync
     • Long term fix is to get rid of the OSUSL sync (in favor or archives.jenkins.io)
     • Changelog is out
   • Ready to deploy on infra.ci

2. Next production operations:

   • Thursday 11 April - 08:00am UTC: infra.ci.jenkins.io data directory (JENKINS_HOME) change from LRS to ZRS volume in Azure (required for migrating controller to arm64)
     • Eventually controller migration to arm64 (WiP - see below)
   • New update center load testing (WiP - see below)
   • ACP Nginx tuning - Thursday, 08:00am UTC

Upcoming Calendar

• Next Weekly: 2.454, Tuesday April 16
• Next LTS: 2.440.3, Wednesday 17 April
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event: cdCon

Cloud Budgets

• Azure (CDF paid)
  • Last month (March): $4,298
  • Current month (April):
    • Consumed: $1,165
    • Forecasted: $4,182
• Azure Sponsorship (Microsoft Credits)
  • Last month (March): $2,375
  • Current month (April): $536
  • Credits left: $32,119 until 31 May 2024
    • Mark check with CDF for extension of this date or renewal
• AWS (CloudBees)
  • Last month (March): $9,567.65
  • Current month (April):
    • Consumed: $2,951
    • Forecasted: $9,692
• AWS (Sponsored by AWS)
  • Credits left: $60,000 (TODO) until 31 January 2025
  • Mark Waite has the action item to submit 2025 donation to AWS given this expiration date
• DigitalOcean
  • Credits left: $17,752.61 until 02 January 2025
  • Last month (March): $939
  • Current month (April):
    • Consumed: $254
    • Forecasted: $850

Notes

• Done:

  • Check if we could replace blobxfer by azcopy
    • Takeaway: way faster ! pkg sync.sh <2mn (instead of 18 to 20mn)
  • ACP:
    • Artifacts proxy does not deliver incremental jars
    • Artifact Caching Proxy (ACP) does not cache artifacts from Maven Central
    • “NoPluginFoundForPrefixException: No plugin found for prefix ‘help’ in the current project and in the plugin groups” when running on ACP
  • User haroldprude submitted a spam comment to issues.jenkins.io
  • Grant docs team permission on jenkins-infra/plugins-wiki-docs
  • Archive Pegdown formatter plugin repository
  • Update-center / crawler root certificate expires the 04 of May (failing job 1 month before)
    • Google shared calendar does not share notifications: it’s per accounts. We need an alternative (bot/GH actions, whatever).
    • Let’s open an hepldesk issue to describe the problem, discuss solution and track implementation. => @dduportal creates the issue.
  • delphix plugin bundles proprietary dependency
  • jenkins.io and plugin-site-issues.jenkins.io are misbehaving on IPv6 with a low MTU
  • [plugin-health-scoring] Configure new job on ci.jenkins.io

• Closed as not planned:

  • Not able to create account
  • Not able to create account
  • Jenkins Security Scan fail due to Artifactory timeout

• Work In Progress:

  • Release of jenkins/* container images on DockerHub errors with HTTP/429
  • [AWS] Migrate ci.jenkins.io EKS clusters from CloudBees AWS account to Jenkins AWS (sponsored) account
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Stéphane and Damien late on the JEP read/suggestion. Need to work on it next milestone
    • Hervé: Performance benchmark, WiP to get a meaningful load test from the pkg VM Apache2 logs (real life production)
  • infra.ci.jenkins.io on arm64 (controller and agents)
    • Next step: migrate infra.ci.jenkins.io controller to arm64
      • 2 steps migration: data volume to ZRS (Thursday), then arm64 migration (a few requirement to finish)
      • arm64 requirement: sizing a new nodepool, thinking with the “non agent” and “non controller” workloads
  • migrate plugin-site-api generate-data from ci.jenkins.io to infra.ci.jenkins.io
    • Low priority
  • Delete Jira TEST project
    • Passed 1 week: let’s close this issue and let Alex to comment/reopen once available as no other actionnables
  • New mirrors in Romania:
    • New Jenkins mirror in Romania by RCS&RDS
      • Pinged about their HTTPS implementation and news
    • New Jenkins mirror in Romania by Hostico
      • WiP on our side to test their FTP credentials (mirrorbits scan)
        • Local k3s test run of mirrorbits but not easy: need redis and arm64 (mac Silicon)
  • Move the jenkins-stats repositories to the jenkins-infra organization
    • Low priority

• ToDo (next milestone) (infra-team-sync-2024-04-16 Milestone · GitHub)

  • Add markdown-formatter plugin to ci.jenkins.io · Issue #4030 · jenkins-infra/helpdesk · GitHub => markdown plugin
    • PHS score is 100%: good thing
    • However JenSec advise is needed before we install it on ci.j
    • assigned to Mark Wait until JenSec feedback is done
  • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our `releases` repository to identify and quarantine unused artifacts · Issue #4034 · jenkins-infra/helpdesk · GitHub => repo audit on releases
    • Let’s communicate to jenkinsci-dev to ask for help
    • Let’s target May (or later) to really start working on it