I’m transitioning from Azure Pipelines, where I encountered build failures due to CPU/memory limitations on Microsoft-hosted agents for my long-running automation project. I run a Maven project with TestNG and have 11 XML files, each containing multiple classes that take over 5 hours to complete. Ideally, I want to run all files in parallel to improve efficiency.
Can Jenkins support this setup effectively? Could someone advise on default CPU/memory capacities for Jenkins agents and any recommended hardware or cloud instances to support such high-demand tasks?
You should run Jenkins (the controller) on a smaller VM (2 CPU/2GIB memory) should be more than enough. When you run Jenkins inside a docker container you probably need not more than 1 GiB of memory when you don’t have many jobs.
Your build jobs should run on agents that are connected to the controller. The java agent process doesn’t need much resources. But the agent should have enough resources for your maven project. How much I can’t tell (I have agents that have 120CPUs and 360GiB memory).
When you say you want to run in parallel can those be executed on different machines? Then you could attach more agents (there are also plugins that allow to dynamically create agents on demand from from cloud providers)
I want to confirm you that previously, I was using Azure Pipelines with Microsoft-hosted agents, configured with 2 core CPU, 7 GB of RAM. Unfortunately, I encountered significant issues with this setup when running my automation jobs. The limited CPU and memory resources led to frequent build failures, especially with long-running tasks. Increasing the number of agents to 6 didn’t fully resolve the issue, as resource constraints persisted.
And yes, My goal is to run the tests in parallel across multiple agents to complete execution faster. I have used 6 pipeline jobs in parallel, but my requirement is to support 11 jobs.
Given this experience, I’m interested in exploring Jenkins with a similar setup but want to ensure the agents can handle the resource demands for our Maven project with TestNG, where each suite runs for over 5 hours. Could you please provide more guidance on configuring Jenkins agents with dynamic scaling in the cloud or other ways to ensure the stability of long-running, resource-intensive jobs?
I hope this message finds you well. I understand you might have a busy schedule, but I wanted to check if you could provide a response to my earlier query.
Your guidance would be greatly appreciated. Please let me know if you need any further details from my side.
I have no experience with setting up clouds in Jenkins as I don’t use them. For the start you can also try to setup static agents, create as many as you need with the size you want and check if this works. Once confirmed it works you could start using dynamic agents provided via one of the cloud plugins.