Infrastructure Team Meeting - September 05, 2023

[dduportal (Damien Duportal)]

2023 09 05 Jenkins Infra Meeting

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• Hai Dotrong

Announcements

1. Weekly:
   • 2.422 is out (WAR, packages and Docker image)
   • Changelog merged and visible
   • Ready to use

Upcoming Calendar

• Next Weekly: 2.423 September 12, 2023
• Next LTS: 2.414.2 September 20, 2023 (RC tomorrow)
• Next Security Release as per jenkinsci-advisories: September 6, 2023 (tomorrow), plugins only
• Next major event:
  • DevOps World tour is coming
  • NYC: Sept. 13/14, 2023
  • Chicago: Sept. 27
  • Santa Clara in Oct.

Notes

• Done:

  • Remove IP restriction on bounce or migrate to VPN
  • VMs: improve command prompts to avoid confusion between services
    • Nice to have: colors in the prompt
  • ATH builds commonly become unresponsive
  • SSL certificate for ci.jenkins.io expires in 23 days
    • Proposal to improve monitoring, helpdesk issue to be created
    • Existing monitors, to be checked/completed:
      • https://github.com/jenkins-infra/datadog/blob/4f9a7a184dc8670359543b3386c85102b78a3b02/datadog-monitors.tf#L113-L118
      • https://github.com/jenkins-infra/datadog/blob/4f9a7a184dc8670359543b3386c85102b78a3b02/synthetics_cijenkinsio.tf
  • Delete the legacy VPN vpn.jenkins.io related resources
  • Plugin build pipeline stopped working
  • Migrate cert.ci.jenkins.io from prod-public to the private network

• Closed as not planned:

  • Trying to set my password using password reset… mails are not received to reset password

• Work In Progress:

  • IntelliJ plugin builds fail on Windows because Gradle cannot cleanup
    • PR proposed with a workaround to contributor, waiting for feedback
    • “workaround” because we are not heavy Gradle users though
  • Matomo github/docker repos
    • Container Image
      • Renamed
      • Cleaned up to have proper versioning and deployment process
      • Need to create the DockerHub repository for the initial deployment
    • WiP on the MySQL managed instance in Azure
  • JDK21: change from nightly builds to weekly ea builds in the infra
    • packer-image is now using JDK21 ea in production (version 1.24.0 of the template)
    • wip on the puppet (jenkins-infra/jenkins-infra), we had to improve our communication loop but now ready to roll
    • Goal: allow tracking with updatecli for JDK21
  • Assess Artifactory bandwidth reduction options
    • Brownout was success
    • Let’s plan for removing repo1 Thurday 7 o september
    • Damien to report in details + plan for 7
    • Let’s add a blog post (Mark?)
    • Proposal to set time of operation the 7 early day working day of european team
    • Go no go with JFrog later today
  • Proposal for application in publick8s to migrate to arm64
    • We can now build Linux multiplatform docker images for our own infra images, using docker bake like the jenkins official images
    • The wiki.jenkins.io is now served with arm64
    • Next step: lest list the remaining services to migrate
  • Remove remnant of the legacy (overlapped) azure virtual networks
    • One last vnet to be removed by MS support
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Success on using a local rsyncd and a remote HTTP in CloudFlare bucket
      • moved rsync to private in Azure AKS but can be public if we add other mirrors somewhere else
    • Rsyncd container image is updated (wip) to harden it
    • WiP on splitting the current helm chart into 3 parts (mirrorbit, httpd server and rsyncd) to allows different deployment topologies
      • 3 subcharts + a “parent” helm
    • We did not receive any response from CloudFlare about sponsoring us as OSS project.
      • Let’s target DigitalOcean mirror instead for the initial deployment:
        • Outbound bandwitdh cheap and we have credits
        • They asked us, as part of the sponsorship, to try their outbound bandwidth
      • Secondary potential mirrors:
        • OVH (sponsorship to bootstrap but the outbound bandwitdh is free)
        • Oracle (we need a VM and a bit more work, but it would be on a US datacenter)
    • Plan the announcement on mailing lists and blog at least 2 weeks before the deployment, as this service will be served from different IP addresses (on allow lists)
  • [publick8s] Ensure high availability of replicated services
    • We want to work on this before adding more arm64 services in publick8s
  • Remove account request field from Jira login page
    • Nothing done
  • LF status page redirect may be cached for too long
    • Nothing done

• New topics:

  • VPN CRL to renew before 20 September
  • [INFRA-3154] Certain emails from Jenkins mailing lists are failing SPF · Issue #2696 · jenkins-infra/helpdesk · GitHub
  • Hai wants to contribute to infra: we need newbie issues
    • We’ll need this to the Hacktoberfest anyway: let’s think about this in this milestone to get him started
    • Mark proposal: Chinese webiste sunset (with redirections to the current english website).
      • Ref. Redirect Chinese pages to English pages and shutdown the Chinese site · Issue #3379 · jenkins-infra/helpdesk · GitHub
    • Updatecli? we need to find untracked dependencies and add issues about this
    • Less infra, more Jenkins: add Debian 12 support to the jenkinsci/packaging (targetting the molecule test harness for packages)
      • See. Add Debian 12 by NotMyFault · Pull Request #406 · jenkinsci/packaging · GitHub

• ToDo (next milestone) (infra-team-sync-2023-09-12 Milestone · GitHub)