Infrastructure Team Meeting - May 14, 2024

[dduportal (Damien Duportal)]

2024 05 14 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)

Announcements

1. Weekly: 2.458

• Release and WAR are deployed
• Packages are deployed (waiting for mirror sync)
• Docker images are being released (tag created, GH release waiting)
• Changelog merged, and being deployed (wait for jenkins.io Fastly’s de-cache)

2. Reminder for tomorrow: new LTS (2.452.1)!

Upcoming Calendar

• Next Weekly: 2.459 next week (21 May)
• Next LTS: 2.452.1, tomorrow (15 May)
• Next Security Release as per jenkinsci-advisories:
• Next major event:
  • Mark Waite tomorrow at Agile Mile

Cloud Budgets

• Azure (CDF paid)
  • February: $4,147.76 (invoice)
  • March: $4,398 (invoice)
  • April: $4,550 (invoice)
  • May (current): $1862 consumed (Forecast at ~4.2k)
• Azure Sponsorship (Microsoft Credits)
  • Global Status:
    • Credits left: $29,573 until 31 August 2024
  • February: $1,992
  • March: $2,375
  • April: $2,041
  • May (current): $897 (Forecast at ~2k)
• DigitalOcean
  • Global Status:
    • Credits left: $16,980.35
  • February: $650
  • March: $938
  • April: $840
  • May (current): $414 (Forecast at ~$840)
• Request submitted for 2025 donation from AWS
  • Mark Waite submitted, answer expected in June/July 2024

Notes

• Done:

  • account login issue
  • perfecto plugin: 401 error on running “mvn release:perform” for user genesisthomas
  • [ci.jenkins.io] Unable to log in randomly
  • can’t access account, can’t reset password
  • [ci.jio/infra.ci.jio] datadog plugin destroyed the build data. Can we stop using it?
  • To host versioned jenkins.io docs on docs.jenkins.io

• Closed as not planned:

  • Jenkins process spawning large number of gparallelworker.exe processes
  • Unable to login to jenkin account
  • Need Approval for account creation

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud

    • 2 distincts services are working, WiP on update_center2 (initial smoke tests work) to deploy data on both fileshares
    • Next steps described in [INFRA-3100] Migrate updates.jenkins.io to another Cloud · Issue #2649 · jenkins-infra/helpdesk · GitHub
      • Finish and merge update center2 script changes
      • Add a 2nd Cloudflare mirror in the US

  • Packer-Images: bug within updatecli to update goss since exclusion of windows 2019 for vsstudio

    • Goss (test framework) issue since the VsCode 2019 integration blocking releases and updates of our agent images
    • Temp. fix to ensure updatecli works but bug on goss (or updatecli YAML parser?)
    • Short term fix: disabling the goss check so we can release updated image this week
    • Middle term: check the alternative updatecli YAML parser to see if it helps?

  • Add .war.asc to get.jenkins.io

    • Still WiP by Mark. Refactoring in progress.

  • [wiki (docker-confluence-data) deprecated updatecli manifest

    • Ref. Update nginx updatecli manifest · Issue #62 · jenkins-infra/docker-confluence-data · GitHub
    • PR opened and merged
    • New updtecli PR opened
    • WiP: review the new PR and close once a new version of the Docker image is deployed to production with Nginx 1.26

  • [ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account

    • New AKS cluster (terraform managed) is up and running
      • No connectivity issues or any bad surprises from Azure Advisor
      • No NSG for the subnet (otherwise blocks the kubelet) because of the new CNI overlay network. Need to check with network policies
    • New cluster under kubernetes management (with admin user)
      • datadog up and running, CI Jenkins agents ready to roll
    • Next steps:
      • add this cluster to ci.jio, verify, and proceed
      • Note: ACP need to run internally (to the cluster: no ingress)
      • Cleanup
      • Profit

  • [infra.ci.jenkins.io / accountapp] Build stuck waiting for jdk17 label

    • Herve’s PR was not working due to default JDK to 11 while 17 required
    • PR changed by using JAVA_HOME in the pipeline (can’t do it in the VM)
    • New issue: tests, using Selenium, is not working in arm64 agent (in current state)
    • Using Intel VM works! PR ok, approved and merged
    • WiP:
      • Check accountapp is properly released (new Docker image)
      • Issue to be closed once new accountapp version deployed to production

  • Delayed actions:

    • Enable 2FA on jenkinsci npm account
      • When Herve is back from holidays
    • [infra.ci.jenkins.io] Abort and remove (immediately) orphaned builds
      • New team member will take this in June/July
    • Replace Blue Ocean in default display URL (or remove the Blue Ocean plugins)
      • infra.ci and weekly.ci done with success
      • Delayed for ci.jio as it failed (Hervé will take this once back from holidays)

  • Not done, still to do:

    • [Artifactory (repo.jenkins-ci.org)] perform a systematic audit of all non-plugin artifacts in our releases repository to identify and quarantine unused artifacts
    • New Jenkins mirror in Romania by RCS&RDS
    • New Jenkins mirror in Romania by Hostico

• New issues:

  • migrate storage from premium to standard for jenkins-infra, jenkins-weekly and jenkins-release => @smerle
    • Weekly.ci will be the first

• ToDo (next milestone) (infra-team-sync-2024-05-21 Milestone · GitHub)