Jenkins setup:
We have the pipeline which uses the images to push it ECR/Nexus in bamboo. We would like to migrate that to Jenkins. Has anyone done that? The infra is setup in EKS.
I’ve never used Bamboo, but we run our Jenkins instances 100% in AWS EKS.
Note that EKS is rather bare so you will need to add several services so that it autoscales properly.
We use Karpenter (from AWS) to provide proper autoscaling of EC2 instances. The controller has an EBS Persistent Volume, and something to be very aware of is that Kubernetes will want to chown every single file each time the PV is mounted. We ended up with 50m outages until controlling chown could be configured but it is still an issue with BottleRocket AMIs due to their SELinux setup.
The agents are triggered on demand through the Kubernetes plugin for jenkins, and in order to reduce the amount of YAML in our jenkinsfiles we created a custom jenkins library (written in groovy-ish) to help with that.
The controller instances are deployed through Helm charts, which we are moving to be managed by ArgoCD,
One mistake to avoid: do not run your EKS cluster for jenkins in multiple AZs, this will cause a significant amount of cross AZ traffic with no uptime benefit whatsoever. The controller’s EBS volume is AZ specific anyways so it is better to pick one AZ and use it. If the AZ where your controller runs is experiencing an outages, you would be in trouble anyways. If some other AZ is experiencing an outage, your pipelines would break randomly in multi-az setup, so you would actually increase your chances of being impacted by an AZ outage.
I’m not saying you should never run anything in Multi-AZ in AWS, just that if you run in multi-az, your software stack should be made AZ aware and prefer communication within the same AZ to avoid unnecessary costs. Some folks push for multiAZ setups without understanding the full requirements to make it beneficial.
I believe AWS has some tutorials on how to run Jenkins in EKS, your TAM might be able to give you pointers, if you have a TAM.