Infrastructure Team Meeting - October 25, 2022

[dduportal (Damien Duportal)]

2022 10 25 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly:

• 2.375 released (WAR/packages)
• Docker image ready
• Last release item to be done later as usual (such as the changelog)

2. Security Update last week (plugins only)
3. Weekly meeting cancelled the 1st of november (2 weeks milestone: next meeting the 8th of Novemebr 2022)

Upcoming Calendar

• Next Weekly: 1st of November
• Next LTS: 2nd of November (2.361.3)
• Next Security Release: N.A.
• Next major event: N.A.

Notes

• Done:

  • Lost access to publish releases for Crowd2 plugin
  • Add Plugin OpenId Connect Authentication to Crowdin
  • SCM link missingfor several plugins
  • jcroall unable to log in to artifactory - incorrect username/password or locked user
  • Upgrade our GHA using deprecated set-output
  • loki installation is broken since September 2022
  • Account recovery for @andytinkham
  • [ci.jenkins.io] collect datadog metrics for ephemeral VMs
  • Artifactory Log-in no longer working
  • Chore: Swap GH permission level
  • Modifications to the Developers group
  • Delete account on accounts.jenkins.io
  • RPM1004: Error retrieving metadata: Not Found
    • User confirmed their problem was solved
  • Downloading Jenkings LTS (msi package) is too slow
    • Problem fixed, clsoing in favor of Jenkins Mirror
  • Can’t get email for the password
    • Closing as no feedback from user

• Work In Progress:

  • Upgrade to Kubernetes 1.23
    • Digital Ocean (2 clusters)
    • AWS EKS (2 clusters) but not easy (CSI volumes issues and then LB issues)
    • Todo: Azure AKS (2 clusters)
    • Broke the new artifact caching proxy on AWS (see below)
  • Artifact downloads failed on agent using repo cache
    • Partially caused by EKS 1.23 upgrade on eks-public. Removed the AWS repo in chore: remove 'aws' artifact-caching-proxy by lemeurherve · Pull Request #504 · jenkins-infra/pipeline-library · GitHub to allow continuing
    • Also, since Friday, some builds failed due to HTTP/504 errors from the caching proxies
    • Next steps:
      • Improve selection of available repo caches in the future (we’ll have to dynamically change environment variables instead of using static code in pipeline-library)
      • We’ll have to enable datadog logs and metrics on ACP to diagnose the HTTP/504
      • Add a fallback capability in the pipeline-library (check if the local repo cache is available otherwise use JFrog), which requires a healthcheck system in the ACP system
      • (Maybe) Set replication of ACP pods to 2
  • Update center json returning 404
    • Nothing done (not enough time)
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
  • Archive a few Jira components
    • Damien to do it because of JIRA admin
  • Windows ACI 11 agent broken: no git found
  • Jenkins Mirror
    • Requester acknowledged: waiting for them
  • Publish pipeline-steps-doc-generator and backend-extension-indexer artifacts to some kind of storage
    • Tools OK, but OOM on one of the builds
    • next step: publish artefacts on reports.jenkins.io (and try new JDK11 to avoid OOM)
  • https://ci.jenkins.io/job/Infra/job/stories/ is not handling PRs
    • Requirements defined: gotta implement (pipeline writing)
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
    • Todo (nothing done yet)
  • Windows agents are soooooooooo slooooooooooooooooooow
    • Back to backlog, unless we have time to work on it
  • Keycloak performance horrific when looking up / modifying users
    • Delayed, waiting for kube 1.23 upgrade being finished
  • https://twitter.com/jenkins_release is many weeks behind
    • Created the helmchart
      • Secret definition
      • Tested on eks-public
    • Then ready to deploy on production

• New/Important:

  • Add reminder about Jira notification email address · Issue #3204 · jenkins-infra/helpdesk · GitHub => no action for infra-team for now (RFE for accountapp)
  • password · Issue #3200 · jenkins-infra/helpdesk · GitHub => added to current milestone + add an issue to add template for account password recovery (as per @lemeurherve 's idea)
  • [From Platform SIG] Remove PPC64 mentions on ci.jenkins.io => added to current milestone
  • (proposal) Add external-dns to the Kubernetes cluster with an ingress (and a loadbalancer) · Issue #3194 · jenkins-infra/helpdesk · GitHub => to be updated with EIP experiment for DNS update
  • Access to npm namespace => added to current milestone
  • Add observability for the build agents
    • Kept in backlog, but might be interesting to build a custom Grafan stack scoped to ci.jenkins.io (hacktoberfest?)

• ToDo (next milestone) (infra-team-sync-2022-11-08 Milestone · GitHub)