Notes available on GitHub
Participants
- @dduportal (Damien Duportal)
- @hlemeur (Hervé Le Meur)
- @MarkEWaite (Mark Waite)
- @Smerle (Stéphane Merle)
Announcement 
- Announcement
- 2.338 release has completed, no issue detected during release process (Mark to run release)
- 2.332.1 LTS release tomorrow with UI improvements, core library updates
- Cathy Chan is the release lead, with Tim Jacomb as release officer
Notes 
-
Testing Github Milestone to gather infra issues for meeting notes
- Current milestone: infra-team-sync-2022-03-08 Milestone · GitHub
- Adding a milestone for each new week, this could be an easy way to get together all issues/subjects we’re dealing with as we go along
- Current process:
- Create a milestone for the current week
- When creating/updating an issue, set its milestone
- When preparing the meeting notes, consult these issues
- During the meeting notes, ???
- After the meeting notes, not sure closing the milestone would add something, TBD
- Limitation, the milestones can concern one repository only
- A: Let’s ensure that we create 1 issue on helpdesk per “top level subject”
- TODO: @lemeurherve
-
infra.ci cron trigger are not working since the 26th of feb
- [infra.ci.jenkins.io] Cron triggered build are not started since 2022-02-26 · Issue #2803 · jenkins-infra/helpdesk · GitHub
- Hotfix: fix(updatecli) do not reset cron pipeline triggers when no cron expression is passed by dduportal · Pull Request #315 · jenkins-infra/pipeline-library · GitHub
- Long term fix: separate our “mono” pipeline into multiple ones, see subpoint of “infra-reports” below
-
Azure Portal management
- Tim underlined Microsoft security defaults article for a “single click to enforce MFA”
- => Damien
- Tim underlined Microsoft security defaults article for a “single click to enforce MFA”
-
Nginx-ingress & cert-manager upgrade
-
Failed deployment on Azure prodpublick8s cluster due to a missing Service Principal
-
- The github-permissions script returned an incomplete report since the Github bot user used for it has its permissions downgraded in December 2021: https://www.jenkins.io/doc/developer/publishing/source-code-hosting/ misses repositories · Issue #2788 · jenkins-infra/helpdesk · GitHub
- The github-permissions script now use a Github App instead of a Github bot user:
- Updated the script to retrieve an Installation Access Token from the Github App private key and identifier: feat: use a Github App instead of a github bot user for permissions-report by lemeurherve · Pull Request #32 · jenkins-infra/infra-reports · GitHub
- The script has its dedicated Github App
jenkins-infra-reports, installed on the jenkinsci github organization with “Metadata: Read-only” as unique permission - A mechanism to generate a new token every once in a while had to be implemented as an Installation Access Token of a Github App expires after 1 hour: fix: pass a new token every once in a while as context to the graphql client to avoid token expiration by lemeurherve · Pull Request #33 · jenkins-infra/infra-reports · GitHub
A shared pipeline to retrieve this kind of access token is currently in progress: [WIP] feat: add getGithubAppAccessToken by lemeurherve · Pull Request #318 · jenkins-infra/pipeline-library · GitHub=> GitHub App authentication support released- Use https://github.com/jenkinsci/github-branch-source-plugin/blob/7f7aa425a9eb650fe7b75574bd8898bbe7f2b37d/docs/github-app.adoc instead of recreating it in a shared pipeline
- We’ll be able to replace github bot user token credentials with this shared pipeline so we won’t have to rewrite scripts or services depending on it, like for example:
- the fork-report script of the same repository
- updatecli
- For scripts/services needing a longer token, they’ll have to be rewrited to be able to generate one on their own from the Github App private key & identifier
- We’ll be able to decommission the Github bot user, and not suffer from changes of its permissions
- The task of separating the pipeline running the 4 scripts of infra-reports in independant pipelines is in progress: Run each report on their own pipeline · Issue #35 · jenkins-infra/infra-reports · GitHub
- Great Guinea pig for the kind of work we intend to do on all other (shared) pipelines in order to separate main tasks from maintenance ones (like updatecli): Updatecli: Use separated pipelines + organization scanning for all updatecli processes in jenkins-infra · Issue #2778 · jenkins-infra/helpdesk · GitHub
- The task of moving this service from trusted.ci.jenkins.io to infra.ci.jenkins.io is in progress: Migrate infra-report from trusted.ci to infra.ci · Issue #2789 · jenkins-infra/helpdesk · GitHub & Migrate this repository to either infra.ci or release.ci · Issue #30 · jenkins-infra/infra-reports · GitHub
-
JFrog incidents
- “No space left on device”
- Several persons alerted us they couldn’t upload their plugins to Artifactory anymore
- Jenkins-infra/JFrog exchange: https://groups.google.com/g/jenkins-infra/c/ZdyYIhlNJQY/m/_8LnZBfaAwAJ
- Daniel Beck deleted around 120Go of data this weekend, but the issue has been resolved since the beginning of this week only as their GC takes a lot of time.
The artifacts that were deleted will be removed from the filesystem only once the full garbage collection runs.
Please go through this article for more information about the Garbage collection: https://jfrog.com/knowledge-base/how-garbagecollection-mechanism-and-strategies-work-in-artifactory-video/.
In SaaS Artifactory instance, the Garbage collection runs for every 4 hours and it is required 20 iterations to run the full Garbage collection.
- Jesse noted yesterday the service is slow again: (Re) Introduce an artifact caching proxy for ci.jenkins.io · Issue #2752 · jenkins-infra/helpdesk · GitHub
- There were several maintenances announced on their status page and newsletter, an issue to reference them has been opened: Email notifications from JFrog Cloud Status · Issue #2806 · jenkins-infra/helpdesk · GitHub
- JFrog will look if it’s possible to extend our maximum capacity (looks like around 5,6To currently)
- Daniel is excluding some Maven coordinates from artifact resolution in repo.jenkins-ci.org so if you currently download tensorflow or stanford NLP language models from our Artifactory, that’s no longer working (there are Java bindings for CUDA and they’re 350MB)
- “No space left on device”
-
Fastly PURGE requests
- The Fastly API requests to purge the cache of individual URLs is now requiring an API token, while they were allowed to anyone before. (Purging a service via the API still requires a token, this hasn’t changed)
- Since the possibility to purge individual URLs without any requirement was convenient, the use of labels to allow purging a service is proposed: Possibility to purge the Fastly cache of a service via a pull request label · Issue #2811 · jenkins-infra/helpdesk · GitHub
- The alternative is to revert the protection put in place on individual URLs purge requests
- The protection has been reverted.
-
Helpdesk, implementation of issues-similarity
- This Github Action comments on issues with links to similar ones if founded: feat: implement issues-similarity-analysis GHA by lemeurherve · Pull Request #2807 · jenkins-infra/helpdesk · GitHub
- The settings might need some tuning, like lowering the threshold, TBD
-
Update security groups with updatecli
- Goal: automaticaly update security groups on Jenkins controllers setup with EC2 agents (retrieve from jenkins-infra/aws)
- updatecli trying to improve findSubMatch to specify the index of the capture (golang) => Feat/capture Adding a parameter for the transformer FindSubMatch by smerle33 · Pull Request #551 · updatecli/updatecli · GitHub wait for release
-
Garbage Collector for cloud resources on Azure => WiP
-
IRC notifs
- TODO: add notifications from infra.ci
-
Core Release Docker image migrated to release.ci
-
Stale subjects?
- Azure AKS clusters
- AccountApp/Keycloak
- Disable anti spam for cert team
- Email on the mailing list (Google groups) are marked as spam
- Alibaba mirror / mirrors not working? - Mirrors of jenkins update is not working · Issue #2787 · jenkins-infra/helpdesk · GitHub => user gave answer, not a mirror issue, but a wider disucssion about mirroring update-center.json