Infrastructure Team Meeting - September 17, 2024

dduportal · September 24, 2024, 8:40am

Attendees

@dduportal (Damien Duportal)
@jayfranco999 (Jay Reddy)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@kmartens27 (Kevin Martens)

Announcements

Jenkins Weekly Releases:
- Last Week: 2.476 succesfully released
  - JobDSL plugin as minor issues: we had to update it which fixed the problem immediately
- This Week: 2.477 started in time
Next LTS baseline will be selected 18 Sep 2024
- Will include Spring Security 6 Update (2.475 or later)
- Will requires JDK17

Upcoming Calendar

Next Weekly: 2024-09-24 - 2.478
Next LTS:
- 2024-10-02 - 2.462.3
- 2024-10-30 - (to be determined tomorrow).1
Next Security Release as per jenkinsci-advisories: N.A.
Upcoming credentials expirations (~3 weeks):
Next major event:
- DevOps World Virtual online September 17, 2024 (online)
  - Jenkins officers and board presentation and Q&A
- CD Mini Summit in Vienna, September 19, 2024 (on site)
  - Thanks to Olivier Vernin for leading
  - Thanks to Bruno Verachten for presenting

Cloud Budgets

Azure (CDF paid)
- June: $4,287 (invoice)
- July: $4,571 (invoice)
- August: $4,552 ($4,452 cost + $100 monthly support)
- September: $2080 consumed (Forecast at ~ $3.7k )
  - LDAP to arm64, 1 less VM! (Migration left over from publicK8s to arm64 · Issue #3837 · jenkins-infra/helpdesk · GitHub)
  - migrate privatek8s to sponsored account → [privatek8s] Migrate AKS cluster to the sponsored subscription · Issue #4250 · jenkins-infra/helpdesk · GitHub
Azure Sponsorship (Microsoft Credits) - Remaining: $51672 ($48328 consumed) until May 2025
- June: $7.3k consumed
- July: $10k consumed
- August: $10.5k consumed
- September: $5718 (Forecast at ~10k)
DigitalOcean - Remaining ~$15k (~4k consumed) until 02 January 2025
- June: $165.32 (invoice)
- July: $176.01 (invoice)
- August $200.08 (invoice)
  - Bandwidth increase (archives.jenkins.io: fallback of get.jenkins.io)
- September: $52 (Forecast at $160)
AWS:
- CloudBees:
  - June: $5,862
  - July: $6.5k
  - August: $6.3k
  - September: $, forecast at k
- Sponsored account
  - Global Status:
    - Credits left: $60,000 until 31 January 2025
  - Untouched
To open an issue describing the “cloud billing plans” for the upcomings 4 months
- Moving ci.jenkins.io out from Azure (credits) to AWS (to consume credits in AWS and allow Azure Credits to be spent until May)
  - Controller is easy to move
  - Agent VMs is easy (packer, then EC2 plugin) both Linux (Intel, ARM64) and Windows are supported
  - Linux container agent is easy => EKS (includes internal ACP)
  - !! Windows container agents: ACI → Kubernetes
  - !! Docker Registry: does AWS ECR support mirroring mode
- Move CloudBees AW remaining VMs to DigitalOcean (to consume credits)

Notes

Done:
Closed as not planned:
- Не могу зайти в Jenkins
Work in Progress:
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  - 3 brownouts were successful (two of 1 hour and one of 24 hours)
    - Post mortem: [INFRA-3100] Migrate updates.jenkins.io to another Cloud · Issue #2649 · jenkins-infra/helpdesk · GitHub
  - WiP on th main user-facing issue discovered during brownout: we have a lot of users still having an HTTP (without TLS) Update Center URL which fails on the new UC (due to HTTPS enforcing)
    - Meeting question: do we “force” user to go to HTTPS?
      - It’s 2024: should be the good path
      - But if we can avoid “surprise breakage” for users, time to communicate properly (admin monitors, blog posts, etc.)
    - Let’s GO / No Go HTTP this Thursday
  - Planning for final migration next meeting
- stats.jenkins.io:
  - “404” pages on stats.jenkins.io served with 200 OK status code
    - Stephane works on fixing the Nginx conf
  - To host stats.jenkins.io GSoC 2024 project in jenkins-infra
    - Last change by Herve did fix most of the broken usages
    - We have to check for usages
    - We have to rollback plugin-site “hotfix” PRs
    - We have to fix the deployment
- GeoIP Database
  - [GeoIP database][subtask] Add a CronJob geoip updater task for mirrorbits
    - On hold (Update Center), need team pair regarding the azcopy part. Cron Job is good
  - [get.jenkins.io, azure.updates.jenkins.io] MaxMind GeoIP Rate Limit hit when redeploying/upgrading mirrorbits chart
- infra-statistics has no new data since July 1st 2024
  - On hold: waiting for KK
- [ci.jenkins.io] Ensure our build agents have enough entropy (rngd, haveged, etc.)
  - Jay works on Azure VM templates (Linux)
- GSoC (on hold):
- Gradle plugin uses a proprietary dependency
  - WiP on Gradle side (Oleg is aware and drives the subject)
- [infra.ci.jenkins.io] Builds stucks due to GH API rate limit
  - On hold
New Issues (to triage):
ToDo (next milestone) (infra-team-sync-2024-09-24 Milestone · GitHub)
- Update permissions for the tinfoil-scan-plugin
- [Updatecli/kubernetes-management] Improve the Maven manifest.

Hacktoberfest

Proposal to remove all hacktoberfest labels so participation would be really explicit and volunteer

Discussion: SIG Infra should not participate unless we find obvious easy issue but we don’t seem to have which won’t have team spending time on it.
Better to help other SIGs in a controlled effort.

Topic	Replies	Views
Infrastructure Team Meeting - September 03, 2024 Infrastructure meeting , sig-infra	7	September 10, 2024
Infrastructure Team Meeting - August 27, 2024 Infrastructure meeting , sig-infra	14	September 2, 2024
Infrastructure Team Meeting - August 13, 2024 Infrastructure meeting , sig-infra	22	August 20, 2024
Infrastructure Team Meeting - July 09, 2024 Infrastructure meeting , sig-infra	5	July 23, 2024
Infrastructure Team Meeting - January 09, 2024 Infrastructure meeting , sig-infra	188	January 11, 2024