Infrastructure Team Meeting - July 25, 2023

dduportal · July 27, 2023, 11:52am

Attendees

@dduportal (Damien Duportal)
@hlemeur (Hervé Le Meur)
@MarkEWaite (Mark Waite)
@smerle33 (Stéphane Merle)
@poddingue (Bruno Verachten)

Announcements

Weekly: no release today as planned (see below)

Upcoming Calendar

Next Weekly:
- 2.416 tomorrow
- 2.417 1st of August
Next LTS:
- 2.401.3 tomorrow
- 2.414.1 on August the 23rd
Next Security Release as per jenkinsci-advisories: tomorrow (ref. https://groups.google.com/g/jenkinsci-advisories/c/2GiSsACaT28)
- status.jenkins.io PR (@mark)
- Healthcheck the platform and report to JenSec team before @dduportal and @smerle
Next major event: N.A.

Notes

Done:
- I am not able to create a new user account
  - Let’s check Datadog’s logs for AccountApp (even if account has been created) → @lemeurherve
- Forum enhancements
- Rename pipeline-log-fluentd-cloudwatch-plugin-developers
- Cannot download the jenkins war file and plugins
- Jira upgrade broke comment sort order remembering and implies an insane default
Closed as not planned:
Work In Progress:
- Upgrade to Kubernetes 1.25
- [INFRA-3100] Migrate updates.jenkins.io to another Cloud
  - Removed all the Oracle Cloud resources
  - New goal is to see if we can use R2 CloudFlare
    - Benefits: China projection, cheap for outbound bandwidth (no egress fees), multiple locations, full S3 compliant
  - Plan: use Apache based service in publick8s serving .htaccess and HTML files generated by trusted.ci + store the Update Center JSON in R2 (with an HTTP redirectin)
    - Would enable HA
    - Read only htdocs means safer service
  - Q: Do we have agreement from Cloudflare to be sponsored as Open Source organization?
    - We are evaluating with the free-tiers and we’ll apply once we are sure we’ll use them
  - Longer term: we can use another “Mirror redirector” (such as mirrorbits, like for get.jenkins.io) with our own mirrors to project on different networks
- Artifact caching proxy is unreliable
  - Closeable: waiting for final confirmation from developers
- Jenkins server is unable to download plugins from the https://updates.jenkins.io
  - User have to contact Aachen university: nothing else from us (except keeping the issue open, and check again 2 weeks from now)
- Issue while creating Jenkins infrastructure account
  - Short term: create their account to unblock (plugin adopter)
  - Long term: let’s check the logs to see if anything is weird with accountapp
- ATH builds commonly become unresponsive
  - spot disabled for 1 week: let’s check the result
  - Let’s work on a GC (VM > 24h to be removed)
- Disallow issue creation in ‘EVENTS’ project type
  - Todo @dduportal
- Jenkins CI failing for jenkins plugin after changes in jenkinsfile to update jenkins.version
  - We need reporter to answer our questions: let’s ping them
- Assess Artifactory bandwidth reduction options
  - Last discussions led to the maven central could be used as “fallback”
  - If it works with the proposed solution from James, next step will be the next step
  - Q: do we need the “password protected” mirror in Artifactory?
    - Yes: in our infra for reliability, but password is hidden inside ACP (not inside ci.jenkins.io
- [ci.jenkins.io] ATH builds failing due to denied outbound requests during tests
  - Closeable
- AWS: decrease cost for Summer 2023
- Matomo github/docker repos
- Ubuntu 22.04 upgrade campaign
  - VM pkg is the next priority
    - update.jenkins.io service by @lemeurherve
    - pkg.origin.jenkins.io by @dduportal
- LF status page redirect may be cached for too long
  - @en3hD3iMRx6_6IXLNY0Rag takes care of it
New items (triage and consider adding the next milestone)
- AKS cluster publick8s: move Public IPs to a distinct Resource Group than the node pools
- The mirrors.jenkins-ci.org is missing some necessary metadata files, which prevents it from being added as an apt/yum repo
  - Comment that it’s blocked by the update/pkg operation, then backlog
- Garbage Collector on the ci-jenkins-io-artifacts S3 storage account
  - Low priority, backlog
- Proposal for application in publick8s to migrate to arm64
- node tolerations and taint for nodepool arm64
- Kubernetes clusters: define infraciadmin SVC account as code
- Remove IP restriction on bounce or migrate to VPN
- @dduportal to creates 3 issues:
  - Move cert.ci to the new private network → Migrate cert.ci.jenkins.io from `prod-public` to the `private` network · Issue #3688 · jenkins-infra/helpdesk · GitHub
  - Delete vpn.jenkins.io resources → Delete the legacy VPN `vpn.jenkins.io` related resources · Issue #3689 · jenkins-infra/helpdesk · GitHub
  - Delete all remnants of the old (overlap) networks → Remove remnant of the legacy (overlapped) azure virtual networks · Issue #3690 · jenkins-infra/helpdesk · GitHub
ToDo (next milestone) (infra-team-sync-2023-08-01 Milestone · GitHub)