Infrastructure Team Meeting - January 31, 2023

[dduportal (Damien Duportal)]

2023 01 31 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly 2.389 in progress, checklist in progress
2. Announcement: Artifact Caching Proxy
3. We’ll be at FOSDEM this week-end but we keep the weekly meeting next Tuesday

Upcoming Calendar

• Next Weekly: 2.390 weekly release next Tuesday (07th February)
• Next LTS: 2.375.3 the wednesday (8th February)
  • Alex Brandes (NotMyFault) is release lead
• Next Security Release as per jenkinsci-advisories: N.C.
• Next major event: FOSDEM this week end :party:

Notes

• Done:

  • Old inbound-agent published as “latest”
  • Valid ssl certificate for trusted.ci.jenkins.io
  • Remove Deprecated plugin “Handlebars” from our instances
  • Artifact caching proxy 504 Gateway time-out errors with the Azure provider
  • Artifact caching failed on DigitalOcean and AWS Kubernetes pods
  • SQL queries to run on Plugin Health Scoring database
  • IRC bot jenkins-admin is offline
  • (no issue) uplink service was down (dashboard not available at least). Restarted the pod and… it just worked!
    • See new issues about updating dependencies below
    • We can’t tell if any telemetry was lost during the incident
    • It seems that it uses a Sentry.io account: we need to get access (Olivier? Tyler? Baptiste Mathus? Daniel?)
    • PostgreSQL Azure Database to be migrated into our “flexible” instance
  • Add “playwright” tool in our agent images
    • Deployed last friday with 0.55.0
  • Agent template 0.56.0 provides updates JDKs

• Work In Progress:

  • renew the signer certificate for jenkins
    • Digicert access confirmed by @en3hD3iMRx6_6IXLNY0Rag . Let’s work on it after the 2.375.3 LTS
    • Another issue opened to document the process in a runbook
    • 29 January says expiration in 60 days so we’re not late
  • Bump the terraform module for AWS EKS (and consequences)
    • Almost finished, but ACP AWS is now working
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
    • We intend to enable ACP for the buildPlugin() by default this week
    • WiP on providing a pipeline library function to start testing on BOM, ATH, Jenkins Core and other “non plugin” maven builds
      • BOM is enormous so should be interesting to see
    • Will add a dedicated issue + send an email for “plugins” ACP activation
      • One last step before: PR https://github.com/jenkins-infra/pipeline-library/pull/552 to provide per-PR opt-out
      • Another opt-out is to update the Jenkinsfile by setting argument artifactCachingProxyEnabled to false
    • ACP metrics:
      • Datadog metrics show resource usage of all container instances precisely to understand the usage
      • Azure Portal show the disk metrics (IOPS)
      • ACP access logs show if request are served by caching, or from Jfrog (with response time from JFrog)
      • We overprovisionned the container and replicated them, but we still have more resources leverages: scaling up, scaling horizontally, increasing cache disk size, switching cache disk to a better IOPS quality level (we have plenty)
    • Timing: once label PR is merged, we send the email on jenkinsci-dev letting 24h before switching to “default ACP”
  • Ensure all GitHub action versions are pinned and tracked
  • AKS: add cluster publick8s
    • Cluster up and running
    • ACP is running (along with side services)
    • On hold until ACP handles way more things (such as BOM)
  • AKS: add cluster privatek8s
    • Same as publick8s: up and running, infra.ci works well now
    • On hold until ACP handles way more things (such as BOM)
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
    • WiP: LDAP HA
    • WiP: writing but on hold because logs from JFrog shows that mirrored repos are not the major problem

• New/TODO (next milestone)

  • Update uplink.jenkins.io project dependencies
  • uplink.jenkins.io: migrate PostgreSQL database to flexible
  • uplink.jenkins.io: Sentry replacement or access
  • Document the code signing certificate renewal process
    • Let’s work on it after the 2.375.3 LTS
  • Ensure removed jenkins.io pages aren’t accessible and indexed anymore
    • Short term: documentation (both jenkins.io and runbook) + clean up this page (to “test” the whole operation 1 time)
    • Long term: automate clean up
  • Frequent PagerDuty Alerts Disk space is below 1GB free win-xxx-yyyyyy
  • Ask Linux Foundation to renew our Jira license