Jenkins setup:
We are running Jenkins 2.528.3
The job in question runs on a single static worker with 1 executor (due to hardware required
only 1 job can run in parallel)
The job is a declarative pipeline
We in the past weeks we have observer the same scenario at least 3 times:
- The worker goes down (disk space problem)
- Build queue of externally triggered items grows (up to say 4000)
- The worker comes back
- Jenkins is extremely slow processing the back log
The following observations have been made:
-
Even if a job completes on a worker (with status SUCCESS or FAILURE)
it remains visible on the /computer/WORKER and it takes a long time to get another
job scheduled. -
There is no thread dump that can be taken on such job
-
When we query the queue it seems to decrease slowly:
jq '.items[]."_class"' <response from the /queue/api/json API> | uniq -c
583 "hudson.model.Queue$BlockedItem"
3173 "hudson.model.Queue$BuildableItem"
21 minutes later:
jq '.items[]."_class"' <response from the /queue/api/json API> | uniq -c
355 "hudson.model.Queue$BlockedItem"
3169 "hudson.model.Queue$BuildableItem"
- The following messages flood the Jenkins log:
Jan 28 13:09:15 vm2611 jenkins[371907]: 2026-01-28 12:09:15.594+0000 [id=904361] WARNING o.j.p.w.s.s.ExecutorStepExecution$PlaceholderTask#getCau
seOfBlockage: Refusing to build ExecutorStepExecution.PlaceholderTask{label=worker-label,context=CpsStepContext[3:node]:ourjob/6#84013} and going to cancel it, even thoug
h it was supposedly stopped already, because associated build is complete
- We have also seen the following messages (not coming very often)
Jan 28 12:39:06 vm2611 jenkins[371907]: 2026-01-28 11:39:06.802+0000 [id=902454] WARNING o.j.p.w.cps.CpsThreadGroup#saveProgramIfPossible: progra
m state save failed
Jan 28 12:39:06 vm2611 jenkins[371907]: java.nio.channels.ClosedByInterruptException
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:200)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:172)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:309)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.<init>(RiverWriter.java:109)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:638)
Jan 28 12:39:06 vm2611 jenkins[371907]: Caused: java.io.IOException: Failed to persist /var/lib/jenkins/jobs/ourjob/jobs/6/builds/84806/program.dat
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:652)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:613)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:596)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:520)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:372)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:302)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$wrap$4(CpsVmExecutorService.java:143)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
Jan 28 12:39:06 vm2611 jenkins[371907]: at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
Jan 28 12:39:06 vm2611 jenkins[371907]: at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
Jan 28 12:39:06 vm2611 jenkins[371907]: at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
Jan 28 12:39:06 vm2611 jenkins[371907]: at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Jan 28 12:39:06 vm2611 jenkins[371907]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:53)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.call(CpsVmExecutorService.java:50)
Jan 28 12:39:06 vm2611 jenkins[371907]: at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
Jan 28 12:39:06 vm2611 jenkins[371907]: at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
Jan 28 12:39:06 vm2611 jenkins[371907]: at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService.lambda$categoryThreadFactory$0(CpsVmExecutorService.java:50)
I do not expect anyone to troubleshoot this for us, but there is something I’d like to understand what is the logic of the queue processing, why is it so slow and how are “blocked items” created and killed?