Infrastructure Team Meeting - October 17, 2023

dduportal · October 18, 2023, 2:22pm

Attendees

@dduportal (Damien Duportal)
@hlemeur (Hervé Le Meur)
@smerle33 (Stéphane Merle)
@poddingue (Bruno Verachten)
@kmartens27 (Kevin Martens)

Announcements

Weekly: 2.428 is out

Release, packages and the Docker image are out
Changelog in progress, soon

VPN VM had outage for the 4th time today

No issue yet (next milestone)
Restart in the Azure Console solved the issue
Increased the VM size to B2s (to have 2 vCPUs), cost 7$ → 30$ monthly
Candidate for ARM64?

Upcoming Calendar

Next Weekly: 2.429 next Tuesday
Next LTS: 2.414.3 tomorrow
Next Security Release as per jenkinsci-advisories:
Next major event:
- DevOps World Santa Clara Oct. 18-19 2023 followed by a Jenkins community gathering
- FOSDEM at Brussels - Feb. 2024 (3-4 Feb.)

Notes

Done:
Closed as not planned:
Work In Progress:
- ftp.belnet.be should be removed from mirrors or a fall-back offered
  - Disabled temporarily to help user (confirmed)
- track the new jdk21 version from adoptium
  - Packer image: need a fix on the provisioning scripts (remove ea)
  - Puppet (jenkins-infra):
    - Same we have to update the template with the new URL
    - Except for s390x which need to stick to the ea until latest is OK
- Trying to create a new jenkins community account
  - Let’s check there a no account with this email and this username as first step
- Packer: GOSS version tracking and moving sanity check to goss
  - Goss is used for asdf and npm now (with the jenkins user running it: end to end testing instead of only “sanity”)
  - Next: migrate more sanity checks on linux to goss (Stéphane)
  - Next: add goss to Windows checks
- Migrate Terraform states from AWS S3 to Azure buckets
  - AWS IAM temp user created for the migration, which is valid only 1 week (to be deleted once finished)
  - WiP on fastly
  - Todo: AWS and digitalocean Terraform projects
- speed up the Docker image library to create/push tags at the same time for both GH and Docker (instead of running additional build)
  - WiP: Still working on unit tests to define the new behavior. Need pairing and time alone
- Upgrade to Kubernetes 1.26
  - Changelog to finish checking
  - Preparing DO and KS
- Proposal for application in publick8s to migrate to arm64
  - Hervé takes the lead, need knowledge sharing with Stéphane
  - Status: rating.jenkins.io and the Nginx private ingress (including our jenkinsciinfra/404 default web backend) of publick8s are in ARM64
  - Next step: shared sessions between Hervé and Stéphane
  - VPN VM to arm64 => Damien
  - issues:
    - matomo must stay on x86
    - falco (agent) crashing on the latest Ubuntu 22.04 kernels => need to upgrade falco. Damien with pairing
- Matomo github/docker repos
  - Hervé volunteers to help for next steps
  - Should we re-evaluate this choice?
    - Google Analytics v4 (need account superadmin: KK)
    - Keep matomo (less Google, need to learn, we can then import gavin data)
    - Other platforms: plausible?
  - Let’s try with matomo a 2nd time:
    - Remove the “no peristence constraint”
    - Use the full bitnami image + chart, and then we’ll see if we need to integrate Gavin’s customizations
- Planning for supported JDK versions in Jenkins Infrastructure
  - JEP for JDK 2+2+2
    - Is “2 months prior a new LTS release through early access” acceptable for infra?
      - We did it for JDK21
      - EA version are usually available 6-8 month before release
      - As the JDK releases are now deterministic (LTS every 2 years), infra team can add calendar event with reminder to check the new JDKs EAs
    - Is “1 month after EOL of a Java release for Jenkins we remove it” acceptable for infra?
      - Example: December 1 of 2024, JDK11 removed from build agents.
      - Breaks plugins not updated but still using JDK11, but infra can’t deliver new security fixes on JDK11
      - Questions:
        
        “What happens to broken plugins? What is the policy?” => these plugins will soon be either “security-advisored” or made incomaptible with the
    - What about the “non LTS” JDKs?
      - Costs due to paralleliez builds by adding non LTS
      - Is it needed on core or plugins? => we don’t have an answer, the current JEP only cover the LTS
      - Unless there is a compelling reason, infra will not provide non-LTS JDK
  - JDK19 deletion:
    - TODO: ci.jenkins.io infra acceptance tests
    - TODO: Windows container agent
    - TODO: cleanup updatecli shell scripts (checking for JDK19)
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  - Knowledge sharing session with Stephane for handover this milestone
  - Status:
    - WiP: Tests of copying to both AWS S3 (Cloudlfare) and Azure file share (mirrorbits reference bucket) from update-center2 trusted agent, trying to have (generation + rsync + aws s3 sync + azcopy) in less than 3 minutes
    - 2:30 to 3:00: is Daniel ok to extend to 5 min the max time limit?
    - Let’s try paralelization
      - First a naive parallel with background task (& suffix and then wait)
      - GNU parallel if it works
      - rclone is not an option alas.:
        
        can’t parellize copy from 1 source to multiple targets
        
        slower for AWS S3 copies
  - Helm chart ingress for the new mirrorbits => Damien
New topics:
- jenkins.io deletion of old pages (add --delete flag to the jenkins.io deployment process)
  - Documentation team did the checks and we are green
  - Need a backup before trying it (to ensure quick rollback if any problem)
  - Proposal: let’s do it Thursday 19 ?
- Retiring the ZH jenkins.io website - Redirect Chinese pages to English pages and shutdown the Chinese site · Issue #3379 · jenkins-infra/helpdesk · GitHub
  - What can Kevin do to help?
  - Let’s pair, and involve Mark next week as well
ToDo (next milestone) (infra-team-sync-2023-10-24 Milestone · GitHub)