Hi all, has anyone here set up Jenkins on AWS EKS? I have been looking at the docs and there are a ton of rabbit holes to go down. Which is the easiest most durable way with the least amount of headaches? Thanks all
Let’s assume the answer is yes (I personally havnt, but have done k8s installs before).
What would be your actual question?
In general for any async environment it’s better to ask the question rather than to ask to ask the question? Just reduces round trip time.
Hi, our team has been trying to install a durable version of Jenkins on EKS. As I see it there are 2 or 3 ways you can go about this. The way I am currently trying is using helm to install Jenkins and the Jenkins operator with virtual docker in docker agents using configuration as code. The problem with this setup is the Jenkins-operator doesn’t support persistent storage, and the backup only runs once a day. 2nd thing is that our org didn’t want to get a enterprise docker hub account so we have to move all the images to ECR because we where hitting our daily pull limit. (makes it harder to maintain - ie upgrades and such ) 3rd, we are using virtual docker agents which is using docker in docker so trying to figure out how to throttle the job queue and executers so the worker nodes don’t crash before the autoscaler has a chance to do its thing is also a problem. I was asking is someone has done an eks deployment of Jenkins that is in production without these problems and if so which way did you do it?.
We started switching our agents to EKS on https://ci.jenkins.io quite recently so maybe we can help if you have specific questions. Looking at your latest post, it seems to me that you are looking for guidance on the following topics
- How to deploy Jenkins on Kubernetes
- How to switch from DockerHub to ECR
- Correctly handle time out when scaling nodes
How to deploy Jenkins on Kubernetes
On the Jenkins infrastructure project, we decided to use the helm chart, so we have a “custom” helm chart which overrides the official one and add our custom resources link and you can find the jcasc configuration that we are using here
I am not very familiar with the Jenkins operator so I can’t help with that one
How to switch from DockerHub to ECR
It’s not clear to me what you are trying to solve but it appears to me that you want to configure your Kubernetes cluster to pull images from a pull-through docker registry cache to avoid DockerHub pull limit. I don’t think that ECR provides that feature out of the box but with a bit of creativity, I am sure you can figure it away.
Correctly handle time out when scaling nodes
You’ll have to play with the configuration but I can share with you the configuration that we are using on ci.jenkins.io. It’s defined on this public repository jenkins-infra/jenkins-infra
Please note that it’s a ruby erb template as we are using Pupppet to manage ci.jenkins.io