Attendees 
- @dduportal (Damien Duportal)
- @jayfranco999 (Jay Reddy)
- @MarkEWaite (Mark Waite)
- @poddingue (Bruno Verachten)
- @kmartens27 (Kevin Martens)
Announcements 
- Jenkins Weekly Releases
- Last week: 2.493 released with no issues.
- This Week: 2.494 started on time today
- Azure Outages impacting Container Instances
- Reminder:
- [Incident] Windows build of plugins don't start on `ci.jenkins.io` · Issue #4490 · jenkins-infra/helpdesk · GitHub
- Scope: only Windows container builds on ci.jenkins.io
- Communication done last week after team meeting (including installation of customize-header plugin)
- We are still in “degraded” mode: the temporary fix (using VM agents instead of containers) stays until migration to AWS => w
- Issues to create:
- Write down procedure to switch labels to VM (without breaking JDK selection) => @dduportal
- Monitoring should alert us when agent allocations starts to fail => @dduportal
- Increase build retention for jenkins-infra-plugin and the Infra/acceptance-tests
- Datadog monitor (for both builds): status? build time?
- See if pattern emerges to trigger alerts
- New impact detected by @en3hD3iMRx6_6IXLNY0Rag on Jenkins Core Tests which times out around 6h. Short term: if no immediate solution disable test until we’ve migrated to AWS anbd then re-analysis.
- Reminder:
- No team meeting the 4 February 2025 (post-FOSDEM)
Upcoming Calendar 
- Next Weekly: 2.495, Tuesday 28 January 2025
- Next LTS: 2.492.1 - (5 Feb 2025) - Jeremie Playout + Mark Waite as release leads
- Damien will be available (train is Tuesday 4 Feb., not 5 as initially stated)
- If we met requirements for Jeremie, we can start early
- Next Security Release as per jenkinsci-advisories:
- Yes: tomorrow (22 January) - https://groups.google.com/g/jenkinsci-advisories/c/Vj6x6B3XCQo - plugins only
- Upcoming credentials expirations (~3 weeks):
- repo.jenkins-ci.org TLS certificate: SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025 · Issue #4477 · jenkins-infra/helpdesk · GitHub
- Damien and Mark still have to sync to see if we can do it without KK
- LE as fallback
- repo.jenkins-ci.org TLS certificate: SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025 · Issue #4477 · jenkins-infra/helpdesk · GitHub
- Next major event:
- Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025
Cloud Budgets
- Azure CDF:
- October: $4,0k (invoice)
- November: $4,3k (invoice)
- December: $4,4k
January: $2,905 (forecast at $4.5k)
- Increase in DNS billing in our zones
- Increase on trusted.ci (not sure why: crawler failures?) agent
- Azure Sponsorship (Microsoft Credits) - Remaining: $24,714 until May 2025
- October: $12,9k consumed
- November: $13k
- December: $9,5k
- January: $7.4k (forecast at $10,8k)
- DigitalOcean - Remaining ~$15k until January 02, 2026
- October: $195.67 (invoice)
- November: $146 (invoice)
- December: $192 (invoice)
- January: $146 (forecast at $216)
- AWS:
- CloudBees:
- October: $6,4k
- November: $3,9k
- December: $540
- January: $344 (forecast at $530)
- Sponsored account (~$58k credits lefts)
- October: $178
- November: $482
- December: $595
- January: $578 (forecast at $890)
- CloudBees:
Notes 
-
Done:
- Support
- Keep platform up to date
- [cert.ci.jenkins.io] Service Principal used by cert.ci.jenkins.io to spawn Azure agents expires on 2025-02-02
- [terraform-aws-sponsorship] updatecli manifest to track terraform module
karpenter
version - [terraform-aws-sponsorship] updatecli manifest to track
terraform provider versions
- [terraform-aws-sponsorship] updatecli manifest for cluster addon versions
- [terraform-aws-sponsorship] updatecli manifest for
irsa_role
module versions
- ci.jenkins.io to AWS
-
- [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
- [ci.jenkins.io] Move controller (VM) to AWS
- AWS CLI and kubeconfig installed
- Next steps:
- Fix certbot installation error
- Set up TLS and hostname aws.ci.jenkins.io (instead of CNAME ci.jenkins.io) for tessting
- [ci.jenkins.io] Move ephemeral Linux containers to AWS
- JCasc config applied to both existing Linux Kubernetes Agents Clouds (default and BOM)
- Karpenter installed and manually tested with success
- Next steps:
- Create Karpenter NodePools and test Pods are started
- Fix controller HTTPS / DNS issues (see controller issue) and validate agents are starting
- Finally, test a plugin build and a BOM build
- [ci.jenkins.io] Move ephemeral VM agents to AWS
- Windows with SSH (as WinRM is too slow), both 2022 and 2019 works well.
- Note: EC2 Fast Lauch (cost/speed)
- ACP (in EKS) is reachable by the VM agents through an internal LB
- Next steps:
- Set up ACP JCasc and validate for both Linux and Windows
- Apply the full JCasc config (with all agent templates)
- Finaly test a few jobs requiring VMs: Docker Linux, Docker Windows, ATH, and a plugin using VMs explicitly (docker-workflow?)
- Nexts steps:
- Set up EKS Windows nodes instead of ACI ([ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS · Issue #4318 · jenkins-infra/helpdesk · GitHub)
- ECR ([ci.jenkins.io] Set up an ECR pull through cache · Issue #4321 · jenkins-infra/helpdesk · GitHub)
- JENKINS_HOME rsync initial copy Azure → AWS
- Plan ETA for migration
- [ci.jenkins.io] Move controller (VM) to AWS
- Support
- [Incident] Windows build of plugins don’t start on
ci.jenkins.io
- Stale until we migrated to AWS (see above)
- Add Jeremie Playout as a Jenkins LTS release lead
- Should only need to add a few users to the release-team GH’s group in jenkins-infra
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Stale: waiting for feedback from KK
- [Incident] Windows build of plugins don’t start on
- Keep platform up to date
- ci.jenkins.io: upgrade datadog plugin from 8.x to 9.x
- Stale (waiting for AWS migration tests to be finished)
- ETA this Thursday?
- [terraform-aws-sponsorship] updatecli manifest for
ami_release_version
used byeks_managed_node_groups
- WiP: @jay blocked by missing IAM permissions, @dduportal to unblock
- SSL certificate for repo.jenkins-ci.org expires 2 Feb 2025
- Stale: Mark and Damien to meet
- ci.jenkins.io: upgrade datadog plugin from 8.x to 9.x
- Artifactory
- Stale (Damien alone), need to send email to developers to start discussion.
- ETA: post fosdem
- Artifactory outdated
maven-metadata.xml
forpublic/com/github/jnr/jnr-posix/
- Foreign releases in public repository
- Monitoring
- Update Center:
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
- Decommisiong staled
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
- [AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
-
New Issues (to triage):
- Deploy
jenkins-protoype
on Netlify- Keep in Stale, need to answer with ETA after FOSDEM and discuss it in FOSDEM
- [Update Center] HTTP/404 on
/current/updates/*.json*
links- Stale
- External user struggling to submit story to stories.jenkins.io
- Stale
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
- [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
- Deploy
-
ToDo (next milestone) (infra-team-sync-2025-01-28 Milestone · GitHub)
- [trusted.ci.jenkins.io] Crawler fails to publish new tools metadata due to an S3 ↔ Cloudflare R2 error
s3:PutObject NotImplemented: STREAMING-UNSIGNED-PAYLOAD-TRAILER not implemented
- Oracle Patch day: if we cannot find a quick solution, we’ll have to perform a one time manual update to ensure Oracle updates are pushed to the tools
- Issues with Gatsby deployment after dependency upgrades via PR for jenkins-infra/stories
- [terraform-aws-sponsorship] updatecli manifest to track karpenter
helm_release
version - [terraform-aws-sponsorship] updatecli manifest to track AWS Load Balancer
helm_release
version - jenkins.io SPF fail
- Support [skip ci] on default branch
- Downgrading a stats.jenkins.io maintainer to having no rights
- Maven Central artifacts are being downloaded from repo.jenkins-ci.org
- [ci.jenkins.io] Set up an ECR pull through cache
- [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
- [trusted.ci.jenkins.io] Crawler fails to publish new tools metadata due to an S3 ↔ Cloudflare R2 error