Jenkins High Availablity

[raghu-d32 (Raghu ND)]

Jenkins setup:
3 node Dell server cluster.
Deployed jenkins using helm with directpv as provisioner of PV.

Hi,
We want to deploy jenkins with high-availability. Is it possible to do so? Any solutions around modifying statefulsets or deployment config?
We also have minio in our cluster. Can we use minio for the distributed file system and form a solution around this?

Thanks,
RD

[dduportal (Damien Duportal)]

Hi @raghu-d32 Jenkins does not support highly available controllers: don’t even try, you’ll end up corrupting the content of your JENKINS_HOME.

There are solutions to be taken to approach such a setup though:

• Ensure that you have a prebuilt image with the plugins (to avoid downloading or upgrading plugins on restarts)
• Ensure that the persistent volume hosting your JENKINS_HOME is fast not only with I/Os but also when mounting/unmounting to achieve a fast (< 1 min) restart of your controller
• Test every plugin a nd core update offline (so any plugin error would not impact your production)
• Add a “customized” reverse proxy to capture webhook incoming requests (to store them and redeliver them) such as GitHub - jenkins-infra/captain-hook (experimental)
• Ensure that you offload the artifacts storage to an object storage (such as Artifact Manager on S3 for instance)
• Ensure that you have a full Jenkins configuration as code system to ensure reproductibility including jobs (and to cover crash recovery)
• Offload logs and metrics collection to another system such as datadog

Of course, this set of “good practises” might not cover all your cases. If you have a dire needs of high availability you can check with CloudBees commercial product: CloudBees High Availability Management<!-- --> | CloudBees CI plugins (spoiler: CloudBees is my employer but I only work with Open Source Jenkins as they pay me to work on the open source infrastructure, not the commercial product)

[raghu-d32 (Raghu ND)]

Hi @dduportal
Thanks for the reply. Will try the specified approaches.

Keeping high availability aside, is there a way to at least make the data persistent across 3 nodes? Using minio as the storage, will we be able to ensure data persistence in any way? Do let me know.

Using minio for persistent volumes when installing Jenkins on K8s with helm

Hi I am trying to setup Jenkins on K8s cluster with Minio providing the persistent volumes (PV). We already have Minio setup on our cluster. I have also created minio tenant with 16 PVs. My helm chart must pick the PVs automatically if integrated with minio tenant. Can someone suggest any ideas on this integration? Setup: OS: Ubuntu 22.04 Jammy K8s dist: kubeadm K8s version: 1.27.02 1 controller, 3 worker nodes Thanks in advance.

Thanks

[dduportal (Damien Duportal)]

@raghu-d32

Keeping high availability aside, is there a way to at least make the data persistent across 3 nodes? Using minio as the storage, will we be able to ensure data persistence in any way? Do let me know.

A given JENKINS_HOME must be only used by a single instance, otherwise the data will be corrupted.

You can absolutely use minIO to store it but:

• the performances will be really bad because it is not a fully compliant POSIX filesystem: not only the performances will be bad but alos you risk having really weird problems: please read Putting a Filesystem on Top of an Object Store is a Bad Idea. Here is why. to have a better understanding of the underlying problem

If you want to set up a disaster recovery for your Jenkins instance, or a passive/active system, you need a way to take/restore snapshots of the underlying filesystem, or perform backup/restores of the directory to/from a location where there is NO I/O write concurency. More details on Jenkins backup/restore in Backing-up/Restoring Jenkins