Participants
Damien Duportal (@dduportal ), Hervé Le Meur (@hlemeur ), Mark Waite (@MarkEWaite ), Tim Jacomb (@timja )
Official minutes on GitHub.
Notes
- Previous weekly Jenkins build failure
- Fixed by updating the Docker image: Release 1.1.0 · jenkins-infra/docker-packaging · GitHub (both tags
1.1.0andlatest) - Should be Ok for the LTS (no changes in the release script)
- For next weekly: Use the same OpenJDK and Maven version as ci.jenkins.io and trusted.ci.jenkins.io · Issue #16 · jenkins-infra/docker-packaging · GitHub
- Fixed by updating the Docker image: Release 1.1.0 · jenkins-infra/docker-packaging · GitHub (both tags
- apt-get upgrade campaign => need for inventory
- ci, trusted, vpn, archives, pkg, updates, rating, bounce, OSUSL machines (was for jira an confluence)
- Checked machines for current status
- could not access census.jenkins.io
- Mark and Damien do not have access
- Damien will send public
- Other VMs?
- Issues:
- rating & bounce: reboot not working, had to change instance type from
t2.microtot2.nano(hypervisor change) - archives: see below (indirect impact)
-
pkg.jenkins.io: frankeinstein-ubuntu (mix of xenial/bionic + packages to remove such as
iw/ `crda)- Fixed with the help fo Mark during the weekly this week
- Weekly release 2.323 delayed while this was repaired
-
usage.jenkins.io: we fell under https://issues.jenkins.io/browse/INFRA-1563: the 500 G volume was not remounted after reboot
- Andrew opened Log in - Jenkins Jira and we fixed by persisting the fstab “manually” until we move to Azure
- Planning to move that service to Kubernetes
- rating & bounce: reboot not working, had to change instance type from
- ci, trusted, vpn, archives, pkg, updates, rating, bounce, OSUSL machines (was for jira an confluence)
- Outages
- jenkins.io, post mortem: https://hackmd.io/VFHwqgZ6QBq-Jt_Dqa42Vg
-
archives.jenkins.io today
- Fixed (issue with the bandwidth module in Apache after kernel upgrade thanks @halkeye)
- Removed the bandwidth limit - need to grant Oracle Cloud access to all infra team memb
- post mortem TODO (@dduportal)
- Kubernetes 1.20 upgrade
- Done, success! Thanks @Ui7sZO86Rq-0GudX4caAvQ (https://issues.jenkins.io/browse/INFRA-3118)
- EKS: We add to manage more resources with Terraform: feat(eks) manage addons through terraform by dduportal · Pull Request #51 · jenkins-infra/aws · GitHub
- Next step: 1.21, improvements in procedure for EKS
- addons
- DNS network driver to allow pods on private networks
- Managed by EKS and Terraform to assure we can use Kubernetes
- DNS network driver to allow pods on private networks
- workers pool upgrades
- put major k8s version in their names
- addons
- Next Kubernetes upgrade 1.20 to 1.21
- TLS cert for repo.jenkins-ci.org
- We should ask KK to handover, WDYT?
- Has JFrog completed the process?
- JFrog says they are ready, need to send certificate to JFrog
- @dduportal
- We should ask KK to handover, WDYT?
- Oracle Cloud:
- Access for Damien and Herve and Tim and Gavin?
- Could we integrate with Azure AD to centralize management?
- Mark work with Tim to grant access to all the right people
- Terraform for More Clouds
- EPIC: https://issues.jenkins.io/browse/INFRA-3135
- WiP on the library https://issues.jenkins.io/browse/INFRA-3132
- Costs
- AWS: 8.7 k€ today, 8.9k€ forecasted (Less than 9k this month !)
- Azure: 8.2k€ today, 8.6 k€ forecasted
- Improvements still to be made, but dramatic savings already
- “updatecli” campaign:
- The Terraform repositories jenkins-infra/aws and jenkins-infra/terraform-states(private) are now tracked by updatecli (both providers and modules)
- todo:
- (WIP) shared pipelines
- “.io” domain renewal:
- The renewal will increase starting in January 1, 2022. For example, with Gandhi it increases from $42 to $55
- Some cost savings available this year
- Gandhi recommends on their website to renew for the maximum of years before the end of the year (up to 9 years renewal in advance)
- Who is the domain registrar for jenkins.io?
- Likely through Tyler Croy
- Herve willing to connect with Tyler on domain renewal
- Who is the domain registrar for jenkins.io?
- The renewal will increase starting in January 1, 2022. For example, with Gandhi it increases from $42 to $55
- fastly-as-code
- Terraform Registry
- Domains are managed by the fastly UI today
- Prefer to manage as code in the future
- @dduportal add to the terraform epic
- Test new solutions, cf https://groups.google.com/g/jenkins-infra/c/k59uSNgPL08/m/4Hx6JQuHBgAJ
- Gradle Enterprise
- Launchable
- Continue discussions email, then discuss next week in meeting
- Karpenter (Introducing Karpenter – An Open-Source High-Performance Kubernetes Cluster Autoscaler | AWS News Blog)
- Not limited to AWS, could make our scaling easier
- Scaleway has a provider-agnostic tool as well