Attendees
- @dduportal (Damien Duportal)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @Stewi
- @kmartens27 (Kevin Martens)
- @poddingue (Bruno Verachten)
- @onenashev (Oleg Nenashev)
Announcements
- Jenkins Weekly Releases
- Last Week: 2.489 released with no hiccups
- This Week: 2.490 started on time
- Announcements:
- 2 weeks breaks:
- No Infra SIG meeting
- No LTS release (next one: 08 Jan 2025)
- (? No weekly releases)
- Mark wants to continue weekly releases, accepting may not happen
- Missing a weekly release might delay detection of issues
- Testing and verification
- Damien, Stephane, Oleg reply to the email thread that we’ve decided to not build weekly during break
- Limited team availability (we are all off, Damien is the fallback in case of emergency)
- 2 weeks breaks:
Upcoming Calendar
- Next Weekly: 2.491, 7 Jan 2025
- Next LTS: 2.479.3, 8 Jan 2025
- Next Security Release as per jenkinsci-advisories: N.A.
- Upcoming credentials expirations (~3 weeks):
- 18 Dec. - stats.jenkins.io Azure SP for production deployment
- 18 Dec. - contributors.jio Azure SP for production deployment
- 18 Dec. - infra.ci.jio Azure SP for production deployment
- 26 Dec. - All Terraform State Azure SP are expiring (8 projects, 8 credentials, automated PRs from https://github.com/jenkins-infra/terraform-states/pull/38 up to https://github.com/jenkins-infra/terraform-states/pull/45)
- Next major event:
- Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025
- Mark has proposed prioritized list of attendees to Jenkins Board, more details this week
- Invited attendees notified this week
- Software in the Public Interest funding travel for some attendees
- CloudBees funding travel for some attendees
- Contributor Summit and FOSDEM, in Brussels, 31 Jan. and 1/2 Feb. 2025
Cloud Budgets
- Azure CDF:
- September: $3.8 (invoice)
- October: $4.0k (invoice)
- November: $4.3k (invoice)
- December: $2.4k (forecast at $4.6k )
- Increase due to outbound bandwidth (~$10 daily additionals)
- Potential $266 monthly saving with the cert.ci, trusted.ci and puppet VM sizing (~$8 daily savings)
- Potential savings by grouping file storages into a single NFS (get.jio + updates.jio + geoipupdate)
- Azure Sponsorship (Microsoft Credits) - Remaining: $35654 ($84346 consumed) until May 2025
- September: $10.4k consumed
- October: $12.9k consumed
- November: $13k
- December: $5.9k (forecast at $10.8k)
- DigitalOcean - Remaining $15,346 ($4k consumed) until January 02 2026 credits extended
- September: $158 (invoice)
- October: $195.67 (invoice)
- November: $146 (invoice)
- December: $101 (forecast at $185)
- AWS:
- CloudBees:
- August: $6.3k
- September: $6.1k
- October: $6.4k
- November: $3.9k
- December: $283 (forecast at $557)
- Sponsored account (~$58k credits lefts)
- October: $178
- November: $482
- December: $246 (forecast at $516)
- CloudBees:
Additional Topics
- Quick not-so-off-topic - JCP meeting on Dec 17 and JEP-14 feedback
- Java Community Process - caring for Java Enhancement Proposals
- Working group that focuses on technical topics
- Tip and Tail - how Java version things, how libraries version things
- Mark to attend session and share Jenkins 2+2+2
- Jenkins and Develocity FOSS sponsorship - is there still potential interest -
- GitHub - gradle/develocity-oss-projects
- “Develocity open source sponsorship for Jenkins CI instances” - thread in jenkins-infra@googlegroups.com - https://groups.google.com/g/jenkins-infra/c/KveP8_0b3zk/m/FjlfcFm2BQAJ
- Develocity - distributed build caching, build scans, predictive test selection (might overlap with CloudBees’ offering being deployed), etc.
- Some first metrics for Jenkins plugins are nice (4x build boost for a Maven based plugin), but a champion is needed to explore further. Oleg has a CoI as a Gradle employee
- The opportunity is still there, not sure about the budget allocation. If there is no follow-up, Gradle will definitely move on.
- Damien: interested in the infra part
- Oleg to follow-up with Damien in January-ish. Some technical details would be needed
- AWS account use for AWS Java SDK v2 work - JENKINS-73638
- Jenkins infra is very interested in the AWS improvements
- Concerned about the EC2 plugin already
- Can allocate one or more developers an IAM profile on a distinct region
- Will need to predict the needed resources for that IAM profile
- What is need to start that process?
- Create a help desk ticket with the list of users and the use case
- Infra will generate the credentials and share them securely with the contributors
- May need a workshop session at Contributor Summit on AWS Java SDK v2
- Create a help desk ticket with the list of users and the use case
- Jenkins infra is very interested in the AWS improvements
Notes
-
Done:
- Support
- Keep platform up to date
- GeoIP Update
-
-
Update Center ([INFRA-3100] Migrate updates.jenkins.io to another Cloud)
- Decommission the old apache vhost and associated data on pkg VM
-
[AWS] Move ci.jenkins.io from Azure (sponsorship) to AWS (sponsorship)
- Good news: AWS support agreed to increase our vCPU quotas (after an initial refusal). We can now have 2000 vCPUs
- [ci.jenkins.io] Create private EKS cluster with “side” services (datadog, ACP, etc.)
- Cluster is up with 1 node group
- Credentials and access is OK (VPN or infra.ci agents)
- WiP on setting up helm releases
- [ci.jenkins.io] Move ephemeral VM agents to AWS
- It works with Puppet full setup!
- Main issue: SSD setup
- Default AMI block storage is used. From gp2 to gp3 => 450 IOPS => 3000 IOPs. But 125 Mb/s
- Slower than Azure VM agents today for reason: Azure uses local NVMe storage
- Require benchmarking, which require… ACP.
- Next: Windows agents
- Next: Spot agents
- [ci.jenkins.io] Move controller (VM) to AWS
- Blocked by agents
-
ACP
- Instability of artifact-caching-proxy
- All errors were Artifactory side
- For the BOM: partial fix using the new “pre-heat” technique. Not generalizable though.
- Client side solution might bee needed, or agent-side
- build failure with useArtifactCachingProxy=true and dependency with version range
- Instability of artifact-caching-proxy
-
Monitoring private builds (Delay to January)
-
Support
- Foreign releases in public repository
- dnf5 update fails with gpgcheck=1
- External user struggling to submit story to stories.jenkins.io
- Infra stats missing since October 2024 data for stats.jenkins.io Plugin Installation Trend feature
- Maven Central artifacts are being downloaded from repo.jenkins-ci.org
-
Keep platform up to date
-
-
New Issues/to triage:
- [Update Center] HTTP/404 on
/current/updates/*.json*
links- Edge case, delayed to January
- (packer-images): Add Win-19, Win-22 (AMD64 and ARM64) AMIs For EC2 agents
- Adding to new milestone
- [ci.jenkins.io] Set up an ECR pull through cache
- Adding to new milestone
- [ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS
- [ci.jenkins.io] Move ephemeral Linux containers to AWS
- Switch agent (java home) to JDK21 default
- Switch default JDK to 21 for pipeline libraries
- Switch default JDK to 21 for build tools
- Move controllers to JDK21 (runtime)
- Move agents to JDK21 (runtime)
- [INFRA-2651] Replace accountapp with (keycloak? Go-authentik? Something Else?)
- [Update Center] HTTP/404 on
-
ToDo (next milestones):