Infrastructure Team Meeting - August 01, 2023

[dduportal (Damien Duportal)]

2023 08 01 Jenkins Infra Meeting

Attendees

• @dduportal (Damien Duportal)
• @hlemeur (Hervé Le Meur)
• @MarkEWaite (Mark Waite)
• @smerle33 (Stéphane Merle)
• @poddingue (Bruno Verachten)
• @kmartens27 (Kevin Martens)

Announcements

1. Weekly: 2.417 is out (packages and the Docker image soon, waiting for the changelo as usual)
2. Core security releases for both LTS and weekly last week
3. Weekly meeting cancelled the 15th August (but still a weekly core release)

Upcoming Calendar

• Next Weekly: 2.418 next week (08 of August)
• Next LTS: 2.414.1 (RC 9th of August) the 23 of August (backports done, one last UI regression reported though)
• Next Security Release as per jenkinsci-advisories: N.A.
• Next major event: N.A.

Notes

• Done:

  • Artifactory 401 error issues:
    • Cannot log into JFrog Artifactory anymore
    • 401 Unauthorized on repo.jenkins-ci.org
    • 401 from repo.jenkins-ci.org
    • unable to auth to artifactory (via UI)
    • 401 from repo.jenkins-ci.org
    • Cannot log into JFrog Artifactory anymore
    • Cannot log into JFrog Artifactory
  • Plugin not found on plugin site
  • Artifact caching proxy is unreliable
  • node tolerations and taint for nodepool arm
  • Jenkins server is unable to download plugins from the https://updates.jenkins.io
  • [ci.jenkins.io] ATH builds failing due to denied outbound requests during tests
  • Upgrade to Kubernetes 1.25
  • Account issues:
    • I am unable to login to issues.jenkins.io and can’t reset my password (email never arrives)
    • Account Issue
    • Issue while creating Jenkins infrastructure account
    • Opened an issue in account-app repo to send a warning if the user email isn’t valid: Add a `WARNING` log when trying to send an email to a user without any valid email registered · Issue #286 · jenkins-infra/account-app · GitHub

• Closed as not planned:

  • Invalid certificate for https://public.publick8s.jenkins.io/

• Work In Progress:

  • [INFRA-3100] Migrate updates.jenkins.io to another Cloud
    • We’re going to use a new instance of our “mirrorbits” helm chart in publick8s to benefit from both HTTP redirection and Apache 2 HTTP webserver
    • Apache is a requirement for the htaccess files part of the Update Center features
    • HTTP redirector will be used for redirecting to CloudFlare hosted R2 buckets (handling the egress download)
  • Proposal for application in publick8s to migrate to arm64
    • javadoc is already in arm64
    • reports.jenkins.io was migrated to arm64 node pool as well
    • Next service to be defined
  • Jenkins CI failing for jenkins plugin after changes in jenkinsfile to update jenkins.version
    • If we cannot find the proper configuration on ci.jenkins.io (and if not a bug on the GH scanning plugin) then we’ll have to document this for developers
  • plugin-site build commonly fails on infra.ci when accessing https://plugins.jenkins.io resulting in a 502
    • Need to keep builds to check the logs
    • Let’s create a datadog monitor for this build to alert us
  • ATH builds commonly become unresponsive
    • We have to measure build success rate without spot, and cost impact
  • Assess Artifactory bandwidth reduction options
    • Wednesday 02 of August, at 14:30 UTC : brownout of 1 hour
    • No more ACP for central artifactory for now
  • Downgrade to HTTP when there is no trailing /
  • VMs: improve command prompts to avoid confusion between services
    • Stephane is volunteer
  • Disallow issue creation in ‘EVENTS’ project type
  • AWS: decrease cost for Summer 2023
  • LF status page redirect may be cached for too long
  • Matomo github/docker repos
  • Ubuntu 22.04 upgrade campaign
  • Remove IP restriction on bounce or migrate to VPN

• ToDo (next milestone) (infra-team-sync-2023-08-08 Milestone · GitHub)