Infrastructure Team Meeting - June 06, 2023

[dduportal (Damien Duportal)]

2023 06 06 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly:
   • 2.408 skipped due to the LDAP outage, 2.409 is being released
2. get.jenkins.io and ldap.jenkins.io will be migrated in the upcoming days (no precise date) with expected outage (will be announced)

Upcoming Calendar

• Next Weekly: 2.410, 13 June 2023
• Next LTS: 2.401.2, RC next week, final release in 3 weeks (28 June 2023)
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event: N.A.

Notes

• Done:

  • ci.jenkins.io repository scan of jenkins.io repo fails with a stack trace
  • Create an ARM64 nodepool on publick8s to start using arm64 pods
    • Be careful of the Azure availability zones!
    • javadoc.jenkins.io is using ARM64 on production \o/
    • Issue to open to list potential candidate for using arm64 in production
  • Can’t create account
  • Remove translation plugin from ci.jenkins.io
  • [PuppetMaster] Migrate VM from OSUOSL to Azure
    • Issue to release lettuce, edamame and radish with disk snapshot to archives
  • Getting an Unauthorized with CD setup with codebuild-cloud plugin
    • Related to trusted.ci migration to Azure
  • ci.jenkins.io plugin BOM agents are not allocated
  • Invite Mostafa into the jenkins-infra/plugin-health-scoring
  • Update controllers to 2.401.1

• Closed as not planned:

  • Can’t login and password reset doesn’t work

• Work In Progress:

  • Jenkins CI failing for jenkins plugin after changes in jenkinsfile to update jenkins.version
    • Let’s watch the result of permission + merge of the (successful) PR.
  • AKS: add cluster publick8s
    • uplink, accountapp and reports were migrated with success
    • Only 4 services remaining to migrate: LDAP, get.jenkins.io, jenkins.io and plugin-site
    • WiP on LDAP
      • outage on the current LDAP due to misconfiguration: fixed
      • new LDAP installation required the file share (sub element of storage account) to be created: fixed in terraform azure
      • New LDAP release to retry and install, then restore an existing dump to check if it works
      • Announcement to be done once end to end test will be done
      • Should not create outage during migration as no write will be done after the last dump/restore
  • [ci.jenkins.io] Use a new VM instance type
    • data disk rsync to do
    • Azure VM agent to inbound in the new subnet (required)
  • Migrate trusted.ci.jenkins.io from AWS to Azure
    • last one mile: update VM agent quota or type for us east 2
  • Ubuntu 22.04 upgrade campaign
    • cert.ci.jenkins.io
  • Upgrade to Kubernetes 1.25
    • back to backlog
  • Install and configure Datadog plugin on ci.jenkins.io
    • Did not had time to work on this
    • Need sync on this. Thursday?
  • Support Linux containers when running on Windows VMs
    • back to backlog
  • Artifact caching proxy is unreliable
  • Please add Jitpack to repositories available when useArtifactCachingProxy=true
    • Proposal of a convention for exception to avoid ACP-caching 3r party repositories
    • Let’s add jitpack as exception on short term
    • 3rd party repository probe for plugin-health-scoring: https://github.com/jenkins-infra/plugin-health-scoring/pull/323
  • Assess Artifactory bandwidth reduction options
    • Proposal of 1st brownout: thursday 8 , during morning USA time
  • Matomo github/docker repos
    • Helm chart requirements to evaluate
    • Assess hardware/storage requirements (issue to update)
      • Ideally use our existing Postgres only MySQL is supported
    • What would be the other alternatives (self hosted / OSS)? Plausible?
      • As we have enough trust in Gavin’s work (he used matomo for the past 18 month for plugin-site, additionally to our Google Analytics) we can proceed
  • Recent plugin BOM releases fail after unexpectedly long run time
    • Blocked by [ci.jenkins.io] Use a new VM instance type · Issue #3535 · jenkins-infra/helpdesk · GitHub
    • Back to backlog (requires ci.jenkins.io migration)

• New issues:

  • [Azure,uplink] Migrate PostgreSQL Single Server to Flexible Server
    • Discovered while trying to migrate uplink to a new DB along with cluster migration
    • Backlog as no emergency
  • Package availability dashboard is empty datadog
    • Most probably consequence of the datadog dashboard cleanupo
    • Adding to next milestone, @lemeurherve volunteers

• ToDo (next milestone) (infra-team-sync-2023-06-13 Milestone · GitHub)