Infrastructure Team Meeting - April 04, 2023

[dduportal (Damien Duportal)]

2023 04 04 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)
• Mukul Kumar
• Sonali Rajput
• @hlemeur (Hervé Le Meur)

Announcements

1. Weekly
   • War file has been built, but no packaging yet (jar file signing failed)
   • Decision pending on next steps (continue current weekly, build new weekly, …)
2. New Digicert Signing Certificate is there and should be ok to use with tomorrow LTS
   • Signs the jar file
   • Signs the MSI installer

Upcoming Calendar

• Next Weekly: depends on needs to assure LTS will build successfully tomorrow
• Next LTS: Tomorrow (revise the Debian stable packaging with new key)
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event:

Notes

• Done:

  • Sunset the robobutler service
  • Valid ssl certificate for cert.ci.jenkins.io
  • [trusted.ci,cert.ci] Add terraform roles and permission for azure on trusted.ci and cert.ci agent resources
  • DigitalOcean: expiration of the API token for jenkins-infra-team account
  • Proving I am not spammer
  • Update social links on GH organizations
  • miss
  • (Re) Introduce an artifact caching proxy for ci.jenkins.io
  • Accounts shows locked status
  • Add https://github.com/jenkinsci/coverage-model to ci.jenkins.io
  • Raising this issue to prove myself as human and get my account verified
  • agent instability
  • Deploy Maven 3.9.1
  • Grant limited access to release.ci to some security team folks
    • Subsequent issue for RBAC restriction on release.ci → @lemeurherve
  • AKS: add cluster privatek8s
  • Delete my demo account(s)
  • Disable PR-merge mode everywhere
  • (2023-03-04) Azure Credential for ci.jenkins.io is expired

• Closed as not planned:

  • i want to create a new account of jenkins
  • Tons of Warning message - WARNING: Skipped parameter {0} as it is undefined on {1}. Set -D{2}=true to allow undefined parameters to be injected as environment variables or -D{3}=[comma-separated list] to whitelist specific parameter names, even though it represents a security breach or -D{2}=false to no longer show this message.

• Work In Progress:

  • Artifact caching proxy is unreliable
    • Not blocked (no ACP when on DO)
    • WiP on progress on enabling ACP DO again
  • GPG key expires on March the 30th
    • Incoming Post Mortem to improve for next time
    • Closed after LTS release tomorrw
  • Digital Ocean: Credits are almost depleted
    • Let’s wait for answer from DO for adding credits
  • Password reset email not coming through
    • We have access to Mailgun now. But not Sendgrid.
    • Wait until end of week, if no sendgrid access then let’s shift to mailgun so we can diagnose such issues
  • Feat(Infra.ci): add Azure ARM64 VMs
    • Updated to Ubuntu 22.04 (as per the issue below), sounds in good direction
    • Waiting for review on the Azure terraform for gallery management
    • Secondary Goal: decrease cloud costs
  • Ubuntu 22.04 upgrade campaign
    • Packer image to rollout to ci.j thursday (testing in infra.ci)
    • Kubernetes node pools (at least Azure, maybe DO)
    • Checking jenkins-infra/docker-*
  • Document the code signing certificate renewal process
    • See below
  • renew the signer certificate for jenkins
    • New one inserted in Azure
    • Need to check:
      • MSI signing
      • Jar signing (by maven)
    • Today’s weekly is released with old cert.
      • Packaging is configured for no MSI at all
    • Alternative 1 - continue packaging as is (no MSI) (current weekly, no MSI)
    • Alternative 2 - add MSI to packaging and new signing (current weekly, includes MSI and signed war file)
      • Validates MSI signing, does not validate war file signing
    • Alternative 3 - new weekly build using the new certificate + MSI back
      • Validates MSI signing and war file signing
    • 1 and 2 are mutually exclusive
    • Let’s go for 2. Then see 3. before LTS tomorrow
    • @dduportal will ask validation to @timja before proceeding, and we’ll let know Kris Stern (as the LTS lead) and Alex Brandes (as a frequent release lead)
  • [INFRA-2754] Realign repo.jenkins-ci.org mission
  • Out of space on a ci.jenkins.io agent in bom build
    • PR to ensure /tmp and /home/jenkins are not overlay

• New issues:

  • Renew update center certificate (crawler and update-center)
  • Migrate trusted.ci.jenkins.io from AWS to Azure
  • [ci.jenkins.io] Azure billing shows huge cloud cost due to outbound bandwidth

• ToDo (next milestone) (infra-team-sync-2023-04-11 Milestone · GitHub)