Why can't my pipeline task be completed?

Using Jenkins Ask a question
1

I’m encountering an intermittent issue where a pipeline passes all stages successfully but hangs indefinitely at the final step:

end of pipeline

image
image577×376 3.06 KB

Normally, this message should immediately be followed by Finished: SUCCESS, but in these cases, the pipeline appears to freeze. Notably:

  1. The build page already shows SUCCESS status (despite the hang), leaving no option to terminate/kill the job—only “Delete” is available.
  2. After deleting one such build, I’m unsure whether pipeline resources (e.g., executors, temp files) were fully released, raising concerns about hidden resource leaks.

Steps to Reproduce (Low Probability):

  • Run pipelines repeatedly; issue occurs sporadically (~1/500 runs).
  • No clear pattern—fails randomly after all steps pass.

Expected Behavior:
After end of pipeline, Jenkins should:

  • Print Finished: SUCCESS promptly.
  • Fully release resources upon build completion/deletion.
3

It seems that someone reported this issue a year ago

I tried upgrading the pipeline-related plugins, but it didn’t work. Recently, it has recurred, about four or five times a week

4

image
image807×396 4.33 KB

Here’s an additional piece of information
The pipeline has ended for a day and the status has returned to “succeed”, but it will still be in the loading state. And based on past experience, as long as this build is not deleted, this loading state will continue and become a zombie task

5

I would suspect some RunListener that is not behaving well in it’s #onCompleted method and hangs.
You might want to look at the threaddump if you see there anything

6

I checked the stack status of Jenkins and found that 1,200 threads were in the TIMED_WAITING state. Is this the problem?

java.lang.Thread.getAllStackTraces().each { 
  thread, stack -> 
    println "\nThread: ${thread.name} (ID: ${thread.id}, State: ${thread.state})"
    stack.each { println "  at $it" } 
}

Thread: Computer.threadPoolForRemoting [#504201] (ID: 26043805, State: TIMED_WAITING)
  at java.base@21.0.2/jdk.internal.misc.Unsafe.park(Native Method)
  at java.base@21.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410)
  at java.base@21.0.2/java.util.concurrent.LinkedTransferQueue$DualNode.await(LinkedTransferQueue.java:452)
  at java.base@21.0.2/java.util.concurrent.SynchronousQueue$Transferer.xferLifo(SynchronousQueue.java:194)
  at java.base@21.0.2/java.util.concurrent.SynchronousQueue.xfer(SynchronousQueue.java:233)
  at java.base@21.0.2/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:336)
  at java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1069)
  at java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
  at java.base@21.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.base@21.0.2/java.lang.Thread.runWith(Thread.java:1596)
  at java.base@21.0.2/java.lang.Thread.run(Thread.java:1583)
7

Hi @mawinter69，I checked the threaddump. The status of “java.lang.Thread.State” is mostly TIMED_WAITING (parking)/TIMED_WAITING (sleeping). I’m not quite sure if this is what you mentioned. Or is there any other way to obtain more information? I still keep the scene of the question

8

image
image464×330 4.02 KB

Help!

The Groovy code in the pipeline has finished executing, and the end of the pipeline is printed, but the task just won’t finish.

I’ve been struggling with this problem for almost a year now, and I still have no clue how to solve it. The problem still occurs intermittently. Does anyone have any suggestions?

9

Similar to the screenshot example, the pipeline hangs for unknown reasons, taking more than 10 minutes to complete. According to user feedback, in severe cases, it can exceed one hour.

10

Last year, when I was trying to troubleshoot the problem, I tried upgrading the Jenkins version and corresponding plugins, upgrading Jenkins from 2.440.3 to 2.504.3, but the problem did not improve.

11

@MarkEWaite

Hello, I’d like to ask why my post about the issue hasn’t received much attention for so long. Is it because my question wasn’t clear enough, or are the Jenkins community members just too busy? As someone who started using Jenkins right after graduation, I usually solve most Jenkins-related problems on my own, and I’m always happy to help others online. But I’m really at a loss with this problem, and it keeps happening. I sincerely hope to get your help, and I also hope that Jenkins and the community will continue to improve.

12

As the support page says:

As an open source project, Jenkins operates on a community-based support model:

  • Community-driven support: Help comes from fellow Jenkins users and contributors, not paid support staff
  • Best-effort basis: Support is provided by volunteers on their own time
  • Self-service oriented: The community provides extensive documentation and resources for self-help
  • No guaranteed response times: Response times depend on volunteer availability and issue complexity
13

I’ve seen this once or twice longer time ago for a single run without taking a deeper look. So it is likely that no-one that is active here has seen that before and was able to find the root cause. Such problems are hard to analyze and it is usually very time consuming.

14

The hang at “end of pipeline” happens because Jenkins waits for all log output streams to close before printing “Finished: SUCCESS”. If anything in your pipeline has spawned a background process that still has its stdout or stderr open, Jenkins blocks indefinitely waiting for that stream to drain. The build shows SUCCESS already because the Groovy execution finished, but the executor thread is stuck on the I/O flush.

The most common trigger is a shell step that runs something in the background:

sh ‘some_command &’

Even with a trailing ampersand, the child process inherits the pipeline’s file descriptors. Until it exits or closes its handles, the pipeline log stream stays open. The fix is to redirect output explicitly:

sh ‘nohup some_command >/dev/null 2>&1 &’

If you are not running explicit background commands, the second cause is a subprocess launched by a tool inside a stage that detaches itself, like a daemon or a cleanup task that was spawned and never waited on. Same fix applies.

To diagnose the specific zombie, take a thread dump while the pipeline is hanging: Manage Jenkins → System Information → Thread Dumps. Look for a thread named something like “Executor #N for…” that is stuck inside LogAction, RunAction, or ProcessTree. That will tell you exactly what file handle it is waiting on.

A few other things worth checking:

If you use parallel blocks, make sure each branch either completes normally or is wrapped in a try/catch. An uncaught exception inside a parallel branch can leave a Groovy CPS thread in a suspended state even after the overall build resolves.

Adding options { disableResume() } to your pipeline options will at least prevent zombie builds from surviving Jenkins restarts if this happens again.

For the build that is already stuck, you can kill it cleanly from the Jenkins Script Console under Manage Jenkins: Jenkins.instance.getItemByFullName(“your-job-name”).getBuildByNumber(N).finish(hudson.model.Result.ABORTED, new java.io.IOException(“Forcing abort”))

That terminates the executor and releases resources without needing to delete the build record.

Hope that helps!

15

Originally, this issue occurred once every one or two months, but recently it’s been happening weekly, affecting all tasks globally and lasting for anywhere from half an hour to an hour.

16

My situation might be a little different from yours. Mine has now escalated to the point where all running pipelines are hanging simultaneously. However, I will try the method you suggested. Thank you very much for your help.

17

Got it, thank you!