Infrastructure Team Meeting - August 13, 2024

[dduportal (Damien Duportal)]

2024 08 13 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @MarkEWaite (Mark Waite)

Announcements

1. Weekly:
   • 2.471 was released succesfully as part of the 2024-08-07 Jenkins Security Advisory
   • 2.472 started on time today
     • OSUOSL is working despite their hardware operation
     • Jetty 12 EE8 is part of this release
2. We had a 2024-08-07 Jenkins Security Advisory last week
   • LTS 2.452.4 and 2.462.1 were released with success, along with plugins and remoting
   • Jenkins infrastructure was up to date less than 24h after the advisory
   • Note: Docker Controller image need a pipeline fix for an edge case (Linux images release on trusted.ci on a LTS line which is NOT the latest LTS) to allow parallel publication of the images. Otherwise, JDK11 images must be done manually (like… last week)

Upcoming Calendar

• Next Weekly: 2.473 - 20 August 2024
• Next LTS: 2.462.2 - Alex Brandes is release lead - 4 September 2024
• Next Security Release as per jenkinsci-advisories: N.A.
• Upcoming credentials expirations (~3 weeks):
  • 2024-08-31: (private link) https://github.com/jenkins-infra/terraform-states/pull/26
  • 2024-08-31: release.ci.jenkins.iuo Azure Client to access secret Vault expires
  • 2024-09-02: trusted.ci.jenkins.io Azure VM Client ID: Extend Azure AD Application password validity on `trusted.ci.jenkins.io` (current end date: 2024-09-02T00:00:00Z) by jenkins-infra-updatecli[bot] · Pull Request #801 · jenkins-infra/azure · GitHub
  • 2024-09-05: DigitalOCean PATs expires
• Next major event:
  • DevOps World Virtual online September 17, 2024
    • Jenkins officers and board presentation and Q&A
  • CD Mini Summit in Vienna, September 19, 2024
    • Thanks to Olivier Vernin for leading
    • Thanks to Bruno Verachten for presenting

Cloud Budgets

• Azure (CDF paid)
  • May: $4,339 (invoice)
  • June: $4,287 (invoice)
  • July: $4,571 (invoice)
  • August: $1845 consumed ( Forecast at ~4.7k)
    • Still need to carefully check why did it increase
    • Public ACP cleanup will decrease a bit
    • Merge publick8s x86 node pools
    • Issues to create to migrate privatek8s (to decrease bill)
    • Issue to create to migrate cert.ci and trusted.ci VMs (3) - (to decrease bill)
    • Issue to create to Use only 1 Redis managed service for both get.jenkins.io and updates.jenkins.io
• Azure Sponsorship (Microsoft Credits) - Remaining: $63633 until May 2025 (instead of August 2024)
  • May: $5k consumed
  • June: $7.3k consumed
  • July: $10k consumed
  • August: $4.3k consumed (Forecast at 10k)
• DigitalOcean - Remaining 16k$ until 02 January 2025
  • May: $648
  • June: $165.32
  • July: $176 consumed
  • August $88 (Forecast at $170)
• AWS:
  • CloudBees:
    • May: $8,281
    • June: $5,862
    • July: $6.5k
    • August: $2.7k, forcast at $6.8k
  • Sponsored account
    • Global Status:
      • Credits left: $60,000 until 31 January 2025
    • Untouched

Notes

• Done:

  • Spam & JIRA Vandalism:
    • Block user coleenwaite for issue spam to jenkins.io
    • Block users jason34 and ruby496williams due to Jira issue spam
    • Block user Osaid due to issue spam
    • Block user michele123 due to issue spam
    • Block user chsonu_512 due to issue spam on issues.jenkins.io
    • Block user chsonu_5 due to issue spam on issues.jenkins.io
    • Spammers
    • Block user rory586 for spam on issues.jenkins.io
    • Block user jennie258fitz due to spam on issues.jenkins.io
    • Block spamming user marlene495hadley
    • Block user ‘browder’ due to spam on issues.jenkins.io
    • Block user ‘adam543’ due to spam comments in Jira issues
  • Add PereBueno to jenkinsci organzation
  • [cert.ci.jenkins.io] Service Principal used by cert.ci.jenkins.io to spawn Azure agents expires on 2024-08-24
  • Update ci.jenkins.io and its friends to Jenkins 2.462.1
  • Disk space is above 90.007% used for agent.trusted.ci.jenkins.io
    • Comes from the new azcopy tasks run in the update-center: their logs were kept but not garbage collected
    • Cleaned up manually once
    • GC implemented
  • get.jenkins.io mirrors
    • temporarily remove xmission from the mirrors list
      • Need to double check datadog monitor for this one (was it checking HTTPS?)
    • New mirror in India
      • Need to add to datadog monitor for early detection
  • Recent contributor spotlight change is not deployed
  • [publick8s] AzureAD / AKS error Authorization Failures have been detected that may affect cluster availability over outbound IPv6 addresses
  • [trusted.ci.jenkins.io] RPU Artifactory API token expires the 2024-08-13
  • VPN access to usage.jenkins.io for usage stats generation
    • Note: KK also need to setup VPN to upload the anonymized data to usage though.
  • Temurin JDK upgrade July 2024
  • Remove 999999-SNAPSHOT version of Remoting from Artifactory
  • Dockerhub rate limit broke the www.jenkins.io CI build
    • Uses ACR (Azure Container Registry). Less burden on the DockerHub, better performances.

• Closed as not planned:

  • not able to sign in
  • jenkins pip脚本问题
  • Activate account
  • Jenkins Account
  • Add jmdesprez to jenkinsci organization
  • I didnt receive an email

• Work in progress:

  • Vandalism in Jira
    • New pattern in JIRA vandalism on the past 8 days
    • Generate a LOT of work for us (see how much issues Mark did treat to block users and revert their changes in JIRA)
    • We had to enable the circuit-breaker on account.jenkins.io, leading to a lot of users issues (unable to create there accounts)
    • After 2 days of circuit breaker, vandalism came back again . So re-enabled for 1-2 days again
    • Short term:
      • Wip on blocking the email adress (as we saw multiple account using consistently the same email). But might not protect us if email changes often. At least for this one.
      • Abuse report sent to Google for this account
      • Let’s also check public IP and eventually block it if it is constant
    • All changes in JIRA have been rollbacked (at least the “sufficient” ones - issue state changes or comments)
    • Medium term: switching away from accounts.jenkins.io (ref. [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?) · Issue #2232 · jenkins-infra/helpdesk · GitHub)
      • Adding user registration steps to make automation more difficult (captcha, email validation, etc.)
  • Cannot signup in jenkins
  • I’ve been blocked
  • Suspend distribution of the windows-slaves-plugin
    • Discussion in progress in the UC PR, going in the right direction
  • Bad gateway message fails some ci.jenkins.io builds
  • [infra.ci.jenkins.io] Builds stuck due to GH API rate limit
    • Next step : create distinct GH apps
  • [Plugin Health Score] Scores not computed - Getting logs from plugin-health.jenkins.io
    • Still on hold (Spring Security work takes Adrien time)
  • Add JDK21 agents (build)
    • Jay is working on Windows inbound and infra.ci
    • Damien works on the SSH Windows agents
  • Migration left over from publicK8s to arm64
    • Still LDAP to reproduce (need to bump versions and OS of the image)
    • Let’s merge the x86 node pools
  • Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra
    • Done azure-net
    • As discussed team-wide: we do NOT want to move existing GHA updateclis to Jenkins (unless it is private Jenkins infra stuff)
    • Scope: Terraform Infra jobs.
  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • Controllers are using it \o/
    • Mark had infinite loop when checking with webbrowser
      • Fixed \o/

• ToDo (next milestone) (infra-team-sync-2024-08-20 Milestone · GitHub)

  • plugin stats for bouncycastle-api are missing data. => looks like a duplicate of 3815