Infrastructure Team Meeting - January 09, 2024

[dduportal (Damien Duportal)]

2024 01 09 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: 2.440

• Build failed after 2h30, due to Github issues
• Build to be retried, released delayed for a few hours

2. @dduportal off next week (16th) and in 3 weeks (30th) and need someone to run the meeting for him

Upcoming Calendar

• Next Weekly: 2.441 next week (2024-01-16)
• Next LTS:
  • RC tomorrow (2024-01-10) 2.426.3
  • LTS happens in 2 weeks (2024-01-24)
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:
  • FOSDEM Brussels, 2-3-4 Feb.
  • Scale: March 15

Notes

• Done:

  • Blocked from creating a second account
  • Artifact caching proxy repo.do.jenkins.io is not answering requests from bom build
    • Forgot to clean ACP cache when changing jfrog config last month
  • Build PR CI check stuck for jenkins-infra/contributor-spotlight
    • We can have other repositories in jenkins-infra with same issue: reopen issue with a comment if the case
  • Split docker-jenkins-weekly and docker-jenkins-weekly.ci for infra.ci and weekly.ci
    • Allows to remove BO from weekly.ci while keeping to infra.ci for instance
    • Docker bake for the win
  • Delete empty GitHub repository https://github.com/jenkinsci/community-gsoc2016-info
  • Remove orphans local repository
  • Approve element.io GH integration app for RPU
  • Add NotMyFault to ‘jenkins-admins’ group
    • ci.jenkins.io only, to allow replaying jobs
  • Replace code-coverage-api with coverage
  • ci.jenkins.io install coverage-badges-extension-plugin
  • crawler build fails because azcopy sync returns a 403
    • Expired credentials from new UC proof of concept (Azure SA, cloudflare API and Cloudflare R2)
    • Renew credentials
    • Restrict IPs allows to access these buckets
      • Since the subscription, we had to crreate addition NAT gateway for outbound fixed IP
      • Azure, no use of outbound IP, only subnet + endpoints. Warning: missing public for updates-jenkins-io
    • Updatecli to detect expiration and open a PR
  • declarative-pipeline-migration-assistant-plugin no longer compiles
    • Another Jfrog operation consquence
    • Updating dependencies helped to remove the problem
  • DNS domain name jenkins.io expires 2024-01-27 (45 days)
  • [get.jenkins.io] migrate from mirrorbits to mirrorbits-parent chart
    • httpd run on arm64
    • 2 downtimes:
      • 8th: TLS issue due to typo (human mistake)
      • 9th: 5 min due to helm chart + network issues
    • httpd is running in read only
  • [publick8s] Tune node pool size
  • [Sponsorships] Renew DigitalOcean sponsorship for 2024

• Closed as not planned:

  • Help with the spam block
  • eee

• Work In Progress:

  • Possible change to ci.jenkins.io’s linux agents causes core test failure
    • Should be closeable, to be confirmed by Alex
  • To host versioned jenkins.io docs on docs.jenkins.io
    • Herve leads (DNS, architecture, etc.)
    • We need a runbook
  • [uplink] Download failing for JavaSystemProperties with error: missing chunk number 0 for toast value xx in pg_toast_xxx
    • PgSql in progress
    • uplink has 300.000.000 records in this table!
  • Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io
  • jgit cloning not converting line ends on windows?
    • Back to backlog
    • No blocker
  • Symbolic link for “latest” for windows-stable in get.jenkins.io points to older release
    • Back to backlog (need azcopy as requirement - below)
  • Check if we could replace blobxfer by azcopy
    • WiP on Azure Service Principal to authenticate with the Azure Storage (as hinted by Tim, SP can be used to generate short-term SAS token allowing access to SA)
    • If not working, then we have to confirm that SAS token can be revoked through expiration date
  • Migration left over from publicK8s to arm64
    • Next: ACP (easy)
    • Then LDAP (need to update it first)
  • Export download mirrors list to a textual representation
    • Textual OK (on reports.jenkins.io)
    • Next step: JSON “API”
      • To add outbound IPs, etc.
  • infra.ci.jenkins.io on arm64 (controller and agents)
    • docker-helmfile done, next step: terraform
  • Packer: GOSS version tracking and moving sanity check to goss
    • No action done, almost there though
  • Redirect Chinese pages to English pages and shutdown the Chinese site
    • Back to backlog (no work to be done this week)
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Waiting for review by JenSec
    • Improved token management (expiration and Ip restriction)
    • Service is not working (rsyncd pod failing to start, due to mount permission)
    • Delayed performance test

• ToDo (next milestone) (infra-team-sync-2024-01-16 Milestone · GitHub)