Infrastructure Team Meeting - May 16, 2023

[dduportal (Damien Duportal)]

2023 05 16 Jenkins Infra Meeting

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)

Announcements

1. Weekly 2.405 release succesfully
   • Changelog published
   • Container image published with new system
   • Few checklist items to be run (as usual) later today
2. Security Advisory, plugins only
   • ci.jenkins.io, trusted.ci and cert.ci updates done
   • docker-jenkins-weekly to deploy
   • docker-jenkins-lts remaining

Upcoming Calendar

• Next Weekly: 2.406 23 May 2023
• Next LTS: 2.401.1 RC tomorrow (17 May), release in 2 weeks (31 May 2023). Alex Brandes (@NotMyFault) is release lead.
• Next Security Release as per jenkinsci-advisories: we had one today
• Next major event: N.A.

Notes

• Done:

  • weekly.jenkins.io migration fixups
  • [ci.jenkins.io] Azure billing shows huge cloud cost due to outbound bandwidth
  • Blocked by Spam Account
  • builds aborting (k8s scaling down?)
  • Add Launchable to agents
  • cant create account
  • Temporary name resolution failures on plugin BOM builds
  • I am not able to create an account
  • Increase disk space for systempool on privatek8s
  • Jenkins CI failing for jenkins plugin after changes in jenkinsfile to update jenkins.version
  • Registered with the wrong email
  • Can’t access applitools account
  • Migrate applications from systempool to linuxpool on privatek8s
  • Credentials for artifactory/maven in lucene-saerch plugin not valid
  • Failed to Deploy Artifacts:… 401 Unauthorized when releasing plugin
  • Jenkins job is failing to load the local jar from lib folder.

• Closed as not planned:

  • password reset

• Work In Progress:

  • Install and configure Datadog plugin on ci.jenkins.io
    • Working on the controller → datadog agent network connection (or socket? wink)
  • [ci.jenkins.io] Use a new VM instance type
  • Migrate trusted.ci.jenkins.io from AWS to Azure
    • 3 VMs created with their network, security groups, disks, etc.
    • VM requested Puppet binding to the puppet master, which is good and expected
    • Next steps: puppet agent run, validate, and start data migration
    • Handed over to Damien because holidays
  • [Azure Billing] Excessive consumption on eastus-cijenkinsio resource group
    • Wait for 2 weeks for billing
    • If spot kills or slows down build too much, we can switch back, but we should move agent network
    • We should move ci.jenkins VM agents to inbound in the new network
  • Feat(Infra.ci): add Azure ARM64 VMs
    • One last mile (updatecli) before closure
    • Done for ci.jenkins.io (@dduportal to check issue and refresh status)
  • AKS: add cluster publick8s
    • weekly.ci.jenkins.io migrated
    • Currently: migrating “stateless” services: javadoc, reports etc.
    • Next “tricky” part: PgSQL based servicesMight need
    • Opportunity to import as code the DNS records
      • DNS TTL to 60s before migration, ideally 24h before a DNS record change
    • V1/V2 for bucket storages: issue to be created
  • DigitalOcean leftover disk to cleanup ?
  • [ci.jenkins.io] Use DigitalOcean VM agents instead of container agents
    • Draft of Packer build for Linux x86_64
    • Next step: try the plugin for Jenkins to spawn VM agents
  • Cleanup and import unmanaged Datadog monitorings in terraform
    • We have some old, deprecated and manually monitors to be cleaned up and moved to IaC
    • We were able to get rid of the custom docker-datadog image \o/
    • Solution to be found for monitoring trusted.ci’s job (update center) as we do not want to use datadog inside the controller itself
  • Artifact caching proxy is unreliable

• ToDo (next milestone) (infra-team-sync-2023-05-23 Milestone · GitHub)