Infrastructure Team Meeting - March 05, 2024

[dduportal (Damien Duportal)]

2024 03 05 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: 2.448

• Packaging OK
• Docker Image is published
• Changelog published

2. Damien off next week: let’s have someone replace him

Upcoming Calendar

• Next Weekly: 2.449 next week (13 March)
• Next LTS: 2.440.2, 20 March 2024 (RC tomorrow)
• Next Security Release as per jenkinsci-advisories: tomorrow (6 march) - https://groups.google.com/g/jenkinsci-advisories/c/x_UkTc9R91s
• Next major event:
  • SCaleX (Los Angeles) in 17-19 March. Alyssa, Basil and Mark will be there

Notes

• Done:

  • Delete Jira bot issues and delete user
  • Improve robustness of CD toolchain
    • Might result in some tokens not updated without us knowing: don’t forget to check last RPU logs if it happens
  • github CD process fails with 401 Unauthorized when deploying to https://repo.jenkins-ci.org/releases
    • Related to above (RPU build failed)
  • Release of jenkinsci/build-monitor-plugin failing with 401 Unauthorized
    • Related to above (RPU build failed)
  • Export download mirrors list to a textual representation
    • New URL: https://reports.jenkins.io/infrastructure/v2/index.json

• Closed as not planned:

  • not able to download jenkins for windows
  • unable to login to page with the credentials used to create

• Work In Progress:

  • Check if we could replace blobxfer by azcopy

    • 3 main repositories (javadoc, jenkinsio and plugin-site) used blobxfer for deployment. All of them has been updated:
      • Storage accounts migrated (in production) to premium (mi. cost is ~20 / month)
      • New authentication (Azure SP + short lived SAS token) used for all
      • azcopy is used
    • Big benefit: upload time drastically decreased!
    • Next steps:
      • jenkins-infra/mirror-scripts (used by the PKG VM)
        • WiP work on Puppet setup to add azcopy and az CLI
        • Team decision to stay on the access key auth. system (as Azure SP would add too much overhead in this context)
      • Cleanups (older storage accounts to remove manually, etc.)

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • azcopy present, but az need to be installed on both agent.trusted and pkg VMs (see “next steps” above)

  • Crawler fails due to lack of authentication

    • SAS renewal done
    • Need to check the Cloudflare (S3) we renewed earlier this morning

  • infra.ci.jenkins.io on arm64 (controller and agents)

    • WiP: Issue with some builds where arm64 agent sizing was too small (OOM killed agents).
      • Rabbit hole: bigger pod means bigger node pools which required a Kubernetes patch upgrade
      • for jenkins.io, let’s use an ARM64 VM to fit with ci.jenkins.io (risk: unavailable images)
      • for account-app, better to check if not the same

  • ftp.halifax.rwth-aachen.de blocked my provider

    • Should be closeable: let’s confirm the user is now unblocked
    • Let’s open a distinct issue about the new mirror provider “Hostico” (:party:)
      • HTTP and sync ok, missing rsync or FTP for us to check in mirrorbits: almost there

  • The UpdateCli step fails regularly when processing jenkins.io PRs

    • To be addressed, not an easy one

  • Exclude JB Space from ACP

    • Alex already opened the PRs: looks good

  • [plugin-health-scoring] Configure new job on ci.jenkins.io

    • WiP, discussion in progress with Adrien. Goal is to decouple plugin site generation from plugin health (static files have better availability)

  • Monitor builds on our private instances (trusted.ci.jenkins.io / infra.ci.jenkins.io / release.ci.jenkins.io)

    • Proposal done (draft) on feat: private instances report by lemeurherve · Pull Request #62 · jenkins-infra/infra-reports · GitHub which generates a complete JSON export
    • To be discussed: the Jenkins user token used to pull information from each controller (lifecycle, permissions on each controller, etc.)
    • Put on hold until we get a security feedback

  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra

  • [INFRA-3125] Migrate jenkins-infra repositories from branch “master” to “main”

• ToDo (next milestone) (infra-team-sync-2024-03-12 Milestone · GitHub)

  • Kube 1.27: Upgrade to Kubernetes 1.27 · Issue #3948 · jenkins-infra/helpdesk · GitHub
    • AWS planned this Thursday (7 March)
    • Azure planned the week of 19 March