Hello,
Is it advisable to run multiple static SSH agents on the same physical or virtual machine?
Could you please help me with the documentation or suggestions? Thanks inadvance.
Regards,
Stalin
Hello Team,
I spent some time reviewing the documentation, but I was unable to locate the relevant information. Could you please provide assistance. Thanks in advance.
Regards,
Stalin
It depends on what you want to accomplish. If your static SSH agents all have the same capabilities (same user account, same permissions, etc.), then multiple static SSH agents do not provide any significant benefit compared to multiple executors on a single static SSH agent. Use a single static SSH agent and increase the number of executors on that agent if needed. A one to one correspondence between an agent and a machine is the most common configuration of static agents.
If your static SSH agents intentionally have different capabilities (different user account or different permissions) at the operating system level, then you probably need a static SSH agent for each of those different capabilities.
Thanks for the details.
Our static agents are configured on monolithic servers (32 CPUs and 256 GB RAM) and are shared across different departments in the organization. To facilitate this sharing, multiple agents are configured on the same server, using the same user and directory.
Recently, we have been encountering issues where some agents fail to reconnect after a Jenkins controller restart, while others occasionally disconnect. These disconnections are causing job failures with RPC errors.
We would like to understand if using monolithic servers as static agents shared by different teams is a best practice. Additionally, how can this setup be improved or implemented more effectively to avoid such issues?
I would not have multiple agents sharing the same user and same directory on a server. I believe that the Jenkins agent assumes it has full control of its assigned directory and writes state information to that directory. Running multiple agents in the same directory seems very likely to confuse the Jenkins agent when state information is written.
In my small test environment, I regularly share a user account for agents between multiple controllers, but I assure that they always use different directories. My test environment is small with no more than 50 agents across many different operating systems and only a few thousand jobs that run for me as a single user.
What I did in the past was run on the same machine 2 agents from one Jenkins instance. They were running in different directories and used different labels. One agent was running builds of a huge C/C++ project (build times of 1 to 2 hours usually but also up to 24h for special builds) and had only a single executor. The other agent was running for small maintenance python script that take usually just a few seconds to minutes without using much CPU, the machine had 80 CPUs and 256 GiB memory and the heavy build uses almost all of it.
Another scenario we’re using is to run agents from 2 different Jenkins instances, one being the productive Jenkins and the other being a test system, again root directories of the agents are different.
A longer time ago we even had agents from 2 different Jenkins instances running the heavy load from above. But we wrote a plugin that ensured that only one heavy build could run on an agent. We used this in an active/passive setup that allowed us to switch the load from one Jenkins to the other (e.g. to perform OS patching or Jenkins upgrades) with zero downtime for our developers (builds were triggered via an external queuing mechanism but also from cron schedules). At that time we still had freestyle jobs, switching the Jenkins was quite some work and we had a check list for the exact steps that need to be done. We no longer do this since we switched to pipeline jobs that survive Jenkins reboots.
So you can run multiple agents on the same machine but you must ensure to use different directories. And you must be aware that you can easily overload the machine if both agents will run heavy work load.
One idea for static agents that must be shared with multiple Jenkins instances might be the Node sharing orchestrator and Node sharing executor plugins. But I’ve never used them so I don’t know how well they work.
Thank you so much for the details.
In our setup, multiple agents are configured on the same server for a Jenkins instance, using the same user but with different root directories.
As a best practice, would you recommend running multiple agents with different users, or is it acceptable to continue with the current configuration?
I also came across a recommendation (not in the official documentation) suggesting the use of different ports for agents. Could you share your thoughts on this?
So you have a single Jenkins and have attached 2 (or even more agents) that run on the same physical machine using different directories. What is unclear is if all the agents have the same number of executors.
Having different users would add some security that a build on one agent is not able to read the workspaces of the other agents (assuming proper setup of the file system permissions regarding groups). But that can be easily bypassed when the developers can modify the assigned agent for freestyle jobs or can freely modify the Jenkinsfile in pipeline jobs.
You machine is rather big so the additional agent processes are not really relevant probably.
From a monitoring perspective it might be better to have only single agent I would say.
To clarify, Jenkins instance is running on a single server (Jenkins controller) and is connected to multiple physical servers as agents. These physical servers are further divided into multiple agents to serve the needs of different departments within the organization.
This setup helps us efficiently share resources across all departments. If we were to configure one agent per server, it would double the number of agents and, at some times, allow one department to occupy the entire server’s resources with their builds.
this is a valid approach