# Parallel stages are being killed **URL:** \ **Category:** Using Jenkins\ **Created:** [October 23, 2024, 10:42am UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296 "2024-10-23T10:42:45Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [October 23, 2024, 10:42am UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/1 "2024-10-23T10:42:46Z")
Hi, we have a Jenkins declarative script that schedules a set of stages across a set of agents using: ```auto parallel p ``` This has worked well, but since early August stages are being randomly killed. The console log shows: ```auto Killed ``` but there is no explanation or reason. How can I determine the reason? Might it be caused by the Jenkins Process Tree Killer? ---
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [October 25, 2024, 7:50am UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/2 "2024-10-25T07:50:34Z")
I have seen in the logs that the jobs are killed because of: ```auto Cannot contact : java.lang.InterruptedException ``` The system log shows: ```auto org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep ... waiting for ... unresponsive for 3 min 0 sec ... org.jenkinsci.plugins.workflow.support.concurrent.Timeout lambda$ping$0 ``` Any suggestions of how to investigate or fix this please? ---
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [October 29, 2024, 12:01pm UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/3 "2024-10-29T12:01:56Z")
I have been advised on Jira that this is likely “a configuration issue on the system that is running the agent. It may be killing the process due to out of memory issues or due to hardware failures or for other reasons that are outside the control of Jenkins.” Does anyone have any advise for how to diagnose this please? ---
**Author:** ![poddingue](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/poddingue/32/985_2.png) [@poddingue](https://community.jenkins.io/u/poddingue)\ **Post date:** [October 31, 2024, 9:41am UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/4 "2024-10-31T09:41:21Z")
The issue you’re experiencing with Jenkins stages being randomly killed could indeed be related to the Jenkins Process Tree Killer or other system-level issues. 🤷 Here are some steps that may help you diagnose and potentially resolve the issue: 🤞 1. Check System Resources: - Monitor the system resources (CPU, memory, disk I/O) on the agents to see if they are running out of resources. - Use tools like `top`, `htop`, or `vmstat` to monitor resource usage. 2. Disable Process Tree Killer: - I’ve heard the [Jenkins Process Tree Killer](https://javadoc.jenkins.io/hudson/util/ProcessKiller.html) could sometimes kill processes that it shouldn’t. Or that was the case a long time ago. 🤷 In any case, you could disable it by setting the `hudson.util.ProcessTreeKiller.disable` system property to `true` (if it still exists, I’m not so sure). Add the following to your Jenkins startup options: `-Dhudson.util.ProcessTreeKiller.disable=true` 3. Increase Logging Level: Increase the logging level for the relevant Jenkins components to get more detailed logs. You can do this in the Jenkins UI under _Manage Jenkins_ → _System Log_ → _Add recorder_. ---
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [October 31, 2024, 6:21pm UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/5 "2024-10-31T18:21:42Z")
@poddingue Thanks very much for your help. Do you have a suggestion for which logger(s) to use? ---
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [November 1, 2024, 5:00pm UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/6 "2024-11-01T17:00:54Z")
The Jenkins declarative script schedules multiple tasks onto an agent, one per available executor. We have a 48 core machine and have 40 executors. When running multiple tasks the machine becomes unresponsive. The Jenkins SSHLauncher times out and jobs are killed. What measures could we take to ensure that the machine remains responsive to SSH? The JVM options for the agent do not specify heap memory limits using `-Xms` and `-Xmx`. The machine has 4GB memory. Should I set heap limits? ---
**Author:** ![mawinter69](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/mawinter69/32/1625_2.png) [@mawinter69](https://community.jenkins.io/u/mawinter69)\ **Post date:** [November 1, 2024, 7:26pm UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/7 "2024-11-01T19:26:17Z")
When you have 48 cores and configure 40 executors then you probably assume that each executors only starts a single threaded process. Many build tool read how many CPUs are available on the machine and then start as many compile jobs. So you if you have such build jobs then you can easily overload your machine. On my Jenkins we start the build of a bigger C/C++ project with cmake and the machine has 120 CPUs but we only have one executor as the build will consume all 120 CPUs and all of the 400GiB of memory. If we would use 2 executors we would need to take extra action to ensure we don’t overload the machine. The whole machine has only 4GiB memory with 48CPU? That is not much. ---
**Author:** ![DavidA2014](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/davida2014/32/88_2.png) [@DavidA2014](https://community.jenkins.io/u/DavidA2014)\ **Post date:** [November 4, 2024, 12:28pm UTC](https://community.jenkins.io/t/parallel-stages-are-being-killed/22296/8 "2024-11-04T12:28:33Z")
@mawinter69 Thanks for your reply. Our builds are single-threaded so each has only one process. I made a stupid mistake checking the system memory - in fact we have 264GB. Should I specify limits for the heap in the agent’s configuration page?