Infrastructure Team Meeting - May 31, 2022

[dduportal (Damien Duportal)]

2022 05 31 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @smerle33 (Stéphane Merle)
• @timja (Tim Jacomb)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly 2.350
   • Delayed (seems to be tests issues)
2. Outages on updates.jenkins.io and pkg.jenkins.io finished (details on jenkins-infra/helpdesk#2888)
3. Update Center Certificate almost expired. No more update center updates since 5 hours ago.

Notes

• Done (infra-team-sync-2022-05-31 Milestone · GitHub)

  • [Datadog] Depreciation of @oncall-* handles in monitors
  • Request access to Jenkins project VPN
  • pkg.jenkins.io certificate expired - varnish cache reports
  • Setup google domain verification for stories.jenkins.io
    TODO: add access to Google console to Mark, Tim, Hervé
  • [update-center] Downloading plugins unstable: Connection timed out
  • Add a GitHub org README

• Work in Progress (infra-team-sync-2022-05-31 Milestone · GitHub)

  • Approve jenkins.io domain on the GH orgs
    • Organization jenkins-infra ok. Waiting for sec team if they want it on jenkins-cert org.
  • [crawler (tools metadata generator)] build is failing on trusted.ci.jenkins.io
    • Top priority, because it also fails the update-center builds.
  • [Infra Docker Images] Deprecate img in favor of docker
    • Draft PR already opened, wip on tests
    • Subsequent issue will be created for removal of img
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    
    • Terraform managed on Oracle Cloud (infra), provisionned with puppet (update center only)
    • Proposal of HA for this service (spread load and improve operations)
    • Once creatred (but not service traffic), have to be included in the update-center JSON publication, along with existing VM
  • [INFRA-3137] Terraform: Import unmanaged Oracle Cloud resources
    
    • No work done (yet). Not able to do it next milestone: delayed.
  • Jenkins Infra Docker Images: build our own Windows images
    • It builds, tests, releases!
    • Had to work around Windows VMs (cygwin, git SCM, credentials)
    • One last fix: git tag per image in “monorepo” (multiple images in same repo) mode
  • Auto notify people based on service routing rules
    • WiP by Herve (side task)
  • [ci.jenkins.io] Digital Ocean sponsorship runs out of credits: stopping the cluster ci.jenkins.io digitalocean
    • Damien and Herve to take appointment with Digital Ocean for next steps
  • [ci.jenkins.io] Container agents in a degraded state
    • To be closed (notes to be published of the post-mortem right before). No more actions from the team.
  • Add an HTTP permanent redirect from https://jenkinsistheway.io to https://stories.jenkins.io/
    • httpredirector deployed in Kubernetes and working (when overriding local DNS)
    • Mark and Alyssa have a confirmed access to the GoDaddy registrar to manage the domain.
    • Short term: they put a redirection in GoDaddy. Google links are not broken anymore BUT redirects to the root of the new website.
    • Medium term: a CNAME should be put to public.aks.jenkins.io to benefit from our own redirector
    • Long term: retrieve the domain name management to the infra-team
    • Removing this issue from next milestone as Mark is Out of Officie this week

• New/Importants infra-team-sync-next

  • DockerHub rate limiting ci.jenkins.io triage
    • Each wave of dependanbot on jenkinsci/docker hit the API rate limit
    • We are waiting from Docker to apply the “team” plan to our organizations so we should have more API rate limit
    • If the improved rate limit is not enough:
      • Agents with public IP and NO docker login when pulling (rate limit per public IP, but for 6 hours)
      • Or a docker registry in caching mode
  • Kubernetes 1.22
    • Stephane to continue upgrading kubectl (update issue with the PR on docker-helmfile)
    • Stephane and Herve to start preparing documentation and migration path: should be upgrade DOKS cluster? Any blockers for others in the changelog?
    • Next week: we’ll determine the timeline if changelog was read as expected
  • Add access to Google console to Mark, Tim & Hervé

• ToDo (next milestone) (infra-team-sync-2022-06-07 Milestone · GitHub)