Infrastructure Team Meeting - July 05, 2022

[dduportal (Damien Duportal)]

2022 07 05 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @smerle33 (Stéphane Merle)
• @MarkEWaite (Mark Waite)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly: on hold.
   • Jetty 10 upgrade has broken websocket agents
     • Apparently no automated test that checks the failure condition
   • Paused the weekly release pipeline while resolving the regression
   • Tim Jacomb is handling the pause and the restart of weekly release

Notes

• Done (infra-team-sync-2022-07-05 Milestone · GitHub)

  • Usual maintenance: 3 issues for GitHub permissions + 1 password reset
    • Password reset method from Mark: if we can associate their Jenkins account with a GitHub account, ask them to fork a jenkins repo to prove they have access to this GH account.
    • Action: Mark add to the Jenkins password reset runbook
  • May JVM statistics not available
    • Resulted in a documentation update: Update documentation for start time by basil · Pull Request #4 · jenkins-infra/jenkins-usage-stats · GitHub
  • CI setup for jenkins-infra/plugin-health-scoring
  • Cannot apt upgrade Jenkins from 2.348 using pkg.jenkins-ci.org
    • TL;DR. HTTP Host Header and SNI must be the same
    • Allowed us to improve puppet: vagrant with docker now

• Work in Progress (infra-team-sync-2022-07-05 Milestone · GitHub)

  • Upgrade to Kubernetes 1.22
    • The 4 clusters are done
    • Post mortem written but to be reviewed
      • Outage on AKS: PVs based on Azurefile buckets (LDAP + get.jenkins.io)
    • Need to write the next upgrade’ issue based on what we learned
    • Need to migrate AKS PV to the new CSI drive
  • Consider removing embeddable-build-status plugin
    • Only ci.jenkins.io left
    • @lemeurherve is working on the batch pull requests as a courtesy to contributors
  • enable Development integration in JIRA
    • Need help from jenkinsci admins.
  • [JENKINS-49707] Evaluate retry conditions to improve the stability of the builds ci.jenkins.io pipeline-library triage
    • Delayed since 2 weeks
    • Need to check if Jesse is there so we can do it today or tomorrow, right afetr the weekly
  • [ci.jenkins.io] Provides both powershell and pwsh on all agent templates ci.jenkins.io
    • Wip: VM OK but still windows container images todo.
  • Migrate updates.jenkins.io to another Cloud
    • WiP: Stephane fighting with network in Oracle. Just solved it 1 hour ago :party: . Cleanup and full PR to create it from scratch and jump to the next step.
    • Next step: puppet \o/
  • Require Java 11 infrastructure thread
    • Moving out of milestone because no jenkins-infra team action expected, but keep an eye on it
  • DockerHub rate limiting
    • Next milestone, it’s on @dduportal (nothing done yet)
  • Replace s390x Ubuntu 18.04 agent with s390x Ubuntu 20.04 agent
    • Next milestone, it’s on @en3hD3iMRx6_6IXLNY0Rag (nothing done yet)
  • ci.jenkins.io agents are very flaky
    • For the next 2 weeks: set of minor changes, hoping to understand what is going on
    • We have to improve observability: traces, aggregating pods logs
  • 502 proxy error when accessing PR view for jenkinsci/jenkins

• New/Important infra-team-sync-next

  • Jfrog slowness: was solved 2 hours after the outage beginning. @dduportal to close it.
  • Downloads /latest directory out of date

• ToDo (next milestone) (infra-team-sync-2022-07-12 Milestone · GitHub)