Ever since we added ThinBackup to the mix a few months back, we’ve been seeing the same thing regularly: a backup was scheduled for the night, but when we come to Jenkins in the morning, the logs mention that it never finished, there is a single pipeline paused in the queue, and there is the “Jenkins is preparing for a shutdown” red plaque.
I have been fiddling with the backups today, and I believe I localized the issue. I ran a pipeline like while(true) { sleep 1 }, and it did not affect the backup in the slightest. Yet once I changed it to node() { while(true) { sleep 1 } }, the backup hanged just right.
(needless to say, we have the Wait until Jenkins is idle to perform a backup setting enabled)
I can see the plugin logging Force quiet mode for jenkins now and wait until all executors are idle, and the source code indicates (if (computer.countBusy() != 0) { running = true;) that it is indeed looking at the executor states, not the job states. This seems extremely wrong to me; the Quiet Mode prevents jobs from completion, and if something is occupying an executor, it will continue to do so indefinitely – clearly causing a deadlock.
The solution to my problem is to simply turn off the quiet mode in the plugin settings, but I am confused. ThinBackup is the backup plugin mentioned by The Book, and is installed on 20k+ instances – yet I cannot find a single mention of this problem, despite the code in question existing for 9+ years. There are JENKINS-46369 and JENKINS-33725 which might’ve have the similar origin, but the issue is so blatantly clear to me, that I’d expect it to happen much more frequently.
Am I really the first to bump into this issue, or am I missing something in my configuration?
Is there anything in the API that the plugin could apply to a running build, instead of checking for the executor state?