Docker agent in stage- taking two executor slots

Using Jenkins Ask a question
1

If I have a stage like this:


pipeline {
    agent any
    stages {
        stage('Test') {
                    agent {
                        docker {
                            image MAVEN_JDK8_DOCKER_IMAGE
                            args DOCKER_ARGS
                        }
                    }

it looks like execution of the stage requires 2 executor slots to run.
Is this correct?

We sometimes run into the situation where multiple pipeline-run-jobs stall each other:
4 executor slots and 4 jobs, each only getting one slot and waiting for the second indefinitly. Is there a way to prevent this?

2

Hello @thigg and welcome to this community. :wave:

Yes, your observation is correct, at least for me. :wink:

In Jenkins Pipeline, when you specify an agent at the top level and then again at the stage level, it requires two executor slots.

The first executor is for the main pipeline job, and the second executor is for the stage that has its own agent directive.

This can indeed lead to a deadlock situation if the number of available executors is less than the number of required executors. :person_shrugging:

In your case, if you have 4 executor slots and 4 jobs, each requiring 2 slots, they can stall each other (been there, done that). :blush:

One way to prevent this is to increase the number of executor slots. However, this might not be feasible if you have resource constraints.

Another way is to restructure your pipeline to use a single agent for the entire pipeline, instead of specifying an agent for each stage.

This would reduce the number of required executor slots to 1 per job.

Here’s an example:

pipeline {
    agent {
        docker {
            image MAVEN_JDK8_DOCKER_IMAGE
            args DOCKER_ARGS
        }
    }
    stages {
        stage('Test') {
            steps {
                // Your steps here
            }
        }
        // Other stages
    }
}

In this example, the Docker agent is specified at the top level, so it’s used for the entire pipeline. This means that each job only requires 1 executor slot. The drawback is that your agent should be able to address all steps, so have all the software installed, whatever the step may be. :person_shrugging:

3

Thanks for your response.

I did not understand before, that the second executor slot was taken by the root agent. This makes much more sense.

I think, I’ll live with the current situation. I was thinking about using a root level agent, but in the end it is more beneficial to have proper resource isolation (cpu-count) for each parallel step.

1 Like