Attendees
- @dduportal (Damien Duportal)
- @MarkEWaite (Mark Waite)
- @smerle33 (Stéphane Merle)
- @poddingue (Bruno Verachten)
- @kmartens27 (Kevin Martens)
- @hlemeur (Hervé Le Meur)
Announcements
-
Last team meeting (2024-03-26) was cancelled due to Update Center outages
- See issue “Check if we could replace
blobxfer
byazcopy
” below for details
- See issue “Check if we could replace
-
Weekly (2.451)
- Available last week
- Package step failed due to Update Center outage? Fixed ~2 hours after the “packaging” job was completed (had to cancel it and run the “sync mirror” steps manually)
-
Weekly (2.452)
- Release and Packages good on first try
- Mirrors are picking the installers: Mirrorlist /windows/2.452/jenkins.msi
- Docker image had issue but is available
- Windows at first try
- Linux failed twice due to Docker HTTP/429 errors, but replays of Linux only solved the problem
- Ready to deploy!
- Changelog: published!
- Release and Packages good on first try
-
DockerHub API Rate limit (HTTP/429)
- We have HTTP/429 since 23 March 2024
- It impairs our ability to deliver Jenkins images (core - weekly, LTS and security releases, and agents)
- Trusted.ci only has 1 outbound IP. Multiple IPs to add for this (short term).
- We need an issue (the problem is not ephemeral)
- 2nd email to Docker OSS program: waiting for an answer until next week
-
Jenkins infra and
xz
CVE- We are not affected on the platform (Ubuntu 18.04 and Ubuntu 22.04)
Upcoming Calendar
- Next Weekly: 2.453, the 9 April 2024
- Next LTS:
- 2.440.3 Release Candidate tomorrow: 3 April 2024
- 2.440.3: 17 April 2024
- Next baseline selection tomorrow (3 april 2024)
- Next Security Release as per jenkinsci-advisories: N.A.
- Next major events:
- cdCon April 16/17/18 (Seattle) with Basil Crow
Cloud Budgets
- Azure (CDF paid)
- January: $6,816.43 (invoice)
- February: $4,147.76 (invoice)
- March: $4,298 (snapshot from costs explorer)
- Azure Sponsorship (Microsoft Credits)
- Global Status:
- Credits used (since the beginning): $7,455
- Credits left: $32,545 (there is an expiration date: to be mentionned next time)
- January: $2,042
- February: $1,992
- March: $2,375
- Global Status:
- AWS (CloudBees)
- January: $9,621.33
- February: $8,556.73
- March: $9,567.65
- AWS (Sponsored by AWS)
- Global Status:
- Credits left: $60,000 (TODO)
- Global Status:
- DigitalOcean
- Global Status:
- Credits left: $17,969.73
- January: $890
- February: $650
- March: $938
- Global Status:
Notes
-
Done:
- [April 2024] Cloudflare API tokens are expired
- Delete comments and spam bot ‘browncheir’
- Create
jenkinsci/build-token-root-plugin-developers
- Junit plugin fails to download and causes Jenkins setup wizard to fail
- Latest commits for Contributor Spotlight not building properly.
- unexpected content in https://get.jenkins.io/plugins/
- Recover Git repository for
remoting-test-client
- Archive
jenkinsci/dom4j
- Delete / block spammer rileynorrisa@outlook.com with username lopun3
- Update ci.jenkins.io, trusted.ci, cert.ci and release.ci to latest LTS version 2.440.2
- Delete comments and spam bot ‘nikolasmurazik’
- Service Principal used by infra.ci.jenkins.io to spawn Azure agents expires on 2024-03-22
- Restrict issue creation in
MEETING
component - Permissions for Jenkins Security Scan repos
- Unexpected delays building small plugin on linux agent
- Revoke an OpenVPN cert for NotMyFault
- [pkg.jenkins.io,releases] Finish cleanup of mirrorbrain
-
-
Check if we could replace
blobxfer
byazcopy
- Update Center outages detailled in the issue
- Next steps:
- Removing blobxfer from docker-builder (PR in progress)
- Removing SAS tokens and some storage accounts
- Announce and list required
-
ACP:
- ACP 1.0.0 featuring server-side fallback to incrementals and Apache Central
- Artifact Caching Proxy (ACP) does not cache artifacts from Maven Central
- Closeable (almost) modulo one last metrics analysis and HTTP/500 errors
- Artifact Caching Proxy (ACP) does not cache artifacts from Maven Central
- “
NoPluginFoundForPrefixException
: No plugin found for prefix ‘help’ in the current project and in the plugin groups” when running on ACP- We have remnants of old Apache Maven plugins in /public that should be moved to a specific custom repo (mostly forks)
- Artifacts proxy does not deliver incremental jars
- On hold: blocked by the issue above
- ACP 1.0.0 featuring server-side fallback to incrementals and Apache Central
-
[INFRA-3100] Migrate updates.jenkins.io to another Cloud
- Stéphane and Damien must read and update the JEP
- Herve works on the performance benchmark
- Need to document/plan (tools, methods) and costs based on existing
- A few side effects from initial benchmark
- We can improve the AKS cluster quality for publick8s (ingress, control plane, syspool nodes)
- We need to start collecting the access logs
- We need to fix the datadog metrics collection for mirrorbits and httpd: both get.jenkins.io and (new) updates.jenkins.io have no metrics
-
- Still todo
-
infra.ci.jenkins.io on
arm64
(controller and agents)- PR (wip) on plugin-site-api
- Next steps:
- Remove docker-hashicorp job on infra.ci (and check others to cleanup if need be)
- Controller on arm64 (need new nodepools?)
-
Add directory HEADER in https://updates.jenkins.io/download/plugins/
- Backlog
-
migrate plugin-site-api generate-data from ci.jenkins.io to infra.ci.jenkins.io
- Damien takes it
-
- We should close this issue as this project is used
-
New mirrors:
- New Jenkins mirror in Romania by RCS&RDS
- Pingging them again
- New Jenkins mirror in Romania by Hostico
- We have FTP for mirror scan: to be tested locally before (to see if FTP credentials are working in mirrorbits)
- New Jenkins mirror in Romania by RCS&RDS
-
Quietly skipped incrementals deployment after retry
- Backlog (postponing 2-3 weeks due to workload)
-
delphix
plugin bundles proprietary dependency- Need to remove old versions (with license issues)
- Driven by the board
-
jenkins.io
andplugin-site-issues.jenkins.io
are misbehaving on IPv6 with a low MTU- Need to work on Letsencrypt renewal using ECDSA
-
[plugin-health-scoring] Configure new job on ci.jenkins.io
- Synced with Adrien, everything good
- Damien need to report in the issue
- Next steps: created CI jobs on ci.j and CD job on infra.ci
-
To host versioned jenkins.io docs on docs.jenkins.io
- On hold
-
-
ToDo (next milestone) (infra-team-sync-2024-04-09 Milestone · GitHub)
- Certificate expiration for
azure.get.jenkins.io
,mirrors.jenkins.io
andmirrors.jenkins-ci.org
(to be confirmed has some were renewed March 26) - Artifactory net issues? (Jenkins Security Scan fail due to Artifactory timeout · Issue #4025 · jenkins-infra/helpdesk · GitHub)
- jenkins-stats to be migrated (Move the jenkins-stats repositories to the jenkins-infra organization · Issue #4017 · jenkins-infra/helpdesk · GitHub)
- Postponed 2 weeks
- Reuse existing homebrew tap of jenkins-infra
- Certificate expiration for
-
JDK22
- Alex Brandes started a new thread about JDK22 for Jenkins
- Should we (infra) provide JDK22 on the ci.j agents?
- Proposal: yes, we add it as a tool but we clearly explain the EOL
- Issue to be created by @lemeurherve