I have the following tests on Jenkins’ pipeline concurrency。
I’m using the controller-agent architecture, and I have a pipeline that includes the pull svn code, and the pull time of a single pipeline is 1 minute.Now I run 100 pipelines simultaneously, all on one agent, and find that the time to pull the svn code has increased to an hour.However, it can be confirmed that the machine resources are sufficient.
So i have the following test. I evenly distribute the 100 pipelines to 10 agents, but all agents are connected to the same machine, that is, each agent executes 10 pipelines at the same time. Test results show that the code pull takes only 20 minutes.
What I want to ask is whether executing multiple pipelines on an agent at the same time will cause performance problems, because it takes only one minute to pull a single SVN, but it takes one hour to pull 100 times on an agent at the same time, which is very unacceptable. If there is this problem, how many pipelines can be executed simultaneously on an agent?
As shown in the following figure, when 100 pipelines are executed concurrently on an agent, the bandwidth for the SVN to pull code is 100 Mbit/s. However, when 100 pipelines are distributed to 10 agents (all connected to the same machine), the SVN pulls 250 Mbit/s bandwidth.
The performance issue you’re experiencing is likely due to the fact that Jenkins agents are not designed to handle a large number of concurrent tasks efficiently. Each pipeline run on an agent is a separate process, and each process requires its own resources. When you run a large number of pipelines concurrently on a single agent, the agent’s resources are divided among all the running pipelines, which can lead to performance degradation.
In your case, when you distribute the pipelines across multiple agents, each agent has fewer pipelines to handle, so it can allocate more resources to each pipeline, resulting in improved performance.
The exact number of pipelines that can be executed simultaneously on an agent depends on various factors, including the hardware specifications of the machine running the agent, the complexity of the pipelines, and the resources required by each pipeline.
However, as a general rule of thumb, it’s recommended to limit the number of concurrent pipelines on a single agent to a number that the agent can handle without significant performance degradation. You might need to perform some tests to determine the optimal number for your specific use case.
As for the bandwidth issue, it’s possible that when you’re running 100 pipelines on a single agent, the network bandwidth is being divided among all the pipelines, resulting in slower SVN pull times. When you distribute the pipelines across multiple agents, each pipeline has more bandwidth available, so the SVN pull operation is faster.
To improve the performance, you might consider increasing the network bandwidth available to the agent, or using a version control system that supports more efficient network operations than SVN.
But here, my different agents are connected to the same machine.That is, I limit the number of pipelines that each agent can run, but the 100 pipelines are still running on a virtual machine.