On customizing agents. Are MLOPS agents good exceptions?

Greetings to all;
I have read in the Jenkins documentation that it is a bad practice to customize Jenkins agents as they are no longer interchangeable and standardized. At the same time there are a few exception in the CD domain.

Now, I’m designing a new jLifeSCI (Jekins in Life Science CI) system where some of the jobs will be making heavy use of Machine Learning and Deep Learning Imaging algorithms and perhaps even use GPUs for processing. This would require the installation of custom Java/Python libraries/packages on these agents, but not necessarily on every agent. Still, a large number of jLifeSCI jobs don’t need them.

So, is this a good case for customizing some agents? Or is it better to set up different controller division strategies where each group of applications (ML vs nonML), gets its own Jenkins controllers.

Thanks for you advice
Ioannis

I think custom hardware (like a GPU) implies that agents are customized, at least a little.

I use the platform labeler plugin to automatically apply labels to agents based on CPU architecture, operating system name, and operating system version. I think you would want to add a label for agents that have a specific GPU so that jobs can request that they run only on agents with that GPU.

That would allow me to label my NVidia Nano single board computer with a label to indicate its GPU.

That doesn’t prevent you from keeping as much of the agent configuration common as you reasonably can, but I don’t see anything wrong with declaring that a job needs specific capabilities (like a GPU or an aarch64 CPU or a riscv CPU). Labels have been enough for my hardware specific needs.

2 Likes

The documentation says:

Agents should not be customized for a particular set of jobs, nor for a particular team.

Lately Jenkins has become more and more popular not only in CI but also in CD, which means that it must orchestrate jobs and pipelines which involve different teams and technical profiles: developers, QA people and Dev-Ops people.

In such a scenario, it might make sense to create customized and dedicated agents: different tools are usually required by different teams

I’m not sure I agree that agents should not be customized. If they shouldn’t then why do they support labels for grouping and classifying them?

We have always had a “general” agent with no tools installed but our shared library bootstraps the agent on the fly. So if we need terraform we install and download the correct version of Terraform using tfswitch. If the job needs python we do the same. But this is slightly slower on the first build of the day. So you could also create images with these tools and use the labels to select them.

You may be interested in my shared library GitHub - DontShaveTheYak/jenkins-std-lib: Bringing the Zen of Python to Jenkins.

It has some pretty notable features like being able to run a sh() step and get the stdout, stderr and combined output and the exit code back. Which isn’t possible in vanilla Jenkins.

It can do other fancy things like run github actions in a Jenkins pipeline =). Real working example jobs are here.

1 Like