How to use environment variables to lock a resource in the options block

Using Jenkins
1

Hello,

I am trying to add the ability to dynamically lock a resource on a multi-branch pipeline before having an agent assigned.

I have multiple stages running in parallel and this is the only stage that requires to have only one instance running per branch. I do not want the stage to take up an agent while it is waiting for the locked resource to become available.

I can hard code the label for my resource but this solution does not work across the multibranch pipeline.

Here is the snippet of my pipeline.

stage('Editor') {
    when {
        beforeAgent true
        beforeOptions true
        allOf {
            expression {
                return params.Stage == "All" || params.Stage == "Editor"
            }
        }
    }
    options {
        lock(resource: "${env.BRANCH_NAME}", inversePrecedence: true)
    }
    agent {
        label 'XYZ'
    }

Is this possible?

2

Hello @CF_Carl and welcome to this community. :wave:

As far as I know, it is possible to lock a resource in a Jenkins multi-branch pipeline dynamically.
You could use the lock step from the Lockable Resources Plugin.
This plugin allows you to lock a resource before the agent is assigned, and it will not take up an agent while it is waiting for the locked resource to become available if I understood correctly.
However, the lock step should be inside the steps block, not in the options block.
The options block is evaluated at the start of the stage, before the when directive, and it doesn’t support variable substitution.

Here is a modified version of your pipeline that could maybe work:

stage('Editor') {
    when {
        beforeAgent true
        beforeOptions true
        allOf {
            expression {
                return params.Stage == "All" || params.Stage == "Editor"
            }
        }
    }
    agent {
        label 'XYZ'
    }
    steps {
        lock(resource: "${env.BRANCH_NAME}", inversePrecedence: true) {
            // Your steps here
        }
    }
}

In this code, the lock step is used inside the steps block. It locks the resource named with the branch name before executing the steps inside the lock block. The inversePrecedence option is set to true to prioritize older builds waiting for the lock.

3

Thank you for your reply.

I have attempted to specify the lock step within the steps of the stage unfortunately this does not behave how I want where the agent is not assigned if the lock is unavailable. In my testing the agent gets assigned and is then blocked from use by other pipelines while it is waiting for the locked resource to become available.

Is it possible to wrap the lock step around the stage?

Or use a different agent while it is waiting for the locked resource? As an example can I have two stages in running sequentially where the first stage checks for the locked resource running on a lower-end agent? Something along the lines of this?

stage("windows") {
    stages {
        agent {
            label "low-end"
        }
        stage("stage 1") {
            steps {
                lock(resource: "${env.BRANCH_NAME}", inversePrecedence: true)
            }
        }
        stage("stage 2") {
            agent {
                label "high-end"
            }
            steps {
                // Perform actual tasks
            }
        }
    }
    post {
        any {
            // Unlock resource??
        }
    }
}
1 Like
4

I think you could use a different agent while waiting for the locked resource. :thinking:
You could use a low-end agent to check for the locked resource and then use a high-end agent to perform the actual tasks.
However, the lock step should be used inside the steps block of the stage where the actual tasks are performed. :thinking:
The lock should be released automatically at the end of the steps block where it is defined, so you wouldn’t need to unlock the resource manually. :crossed_fingers:

stage("windows") {
    stages {
        stage("stage 1") {
            agent {
                label "low-end"
            }
            steps {
                script {
                    def resourceLocked = false
                    try {
                        lock(resource: "${env.BRANCH_NAME}", inversePrecedence: true, skipIfLocked: true) {
                            resourceLocked = true
                        }
                    } catch (e) {
                        // Handle exception if needed
                    }
                    if (!resourceLocked) {
                        error("Resource is locked")
                    }
                }
            }
        }
        stage("stage 2") {
            agent {
                label "high-end"
            }
            steps {
                lock(resource: "${env.BRANCH_NAME}", inversePrecedence: true) {
                    // Perform actual tasks
                }
            }
        }
    }
}

In this code, the first stage (“stage 1”) uses a low-end agent to check if the resource is locked.
If it is locked, Jenkins should throw an error and stop the pipeline.
If the resource is not locked, Jenkins should proceed to the next stage (“stage 2”), where it will lock the resource and perform the actual tasks using a high-end agent.
The lock should be released automatically at the end of the “stage 2” steps block.

5

This should do the trick. It will not assign an agent to the pipeline or stage but explicitly uses the node step inside the lock step.

pipeline {
    agent none
    stages {
      stage('build') {
        steps {
            lock(resource: "${env.BRANCH_NAME}") {
                node("XYZ") {
                    sh "sleep 60"
                }
            }
        }
      }
    }
}
6

Thank you this does produce the desired output where the agent is not assigned until the resource becomes available. This did however bring up a consequence of doing it this way where I was performing some actions in the post block. The post block no longer has an agent to execute on.

pipeline {
    agent none
    stages {
      stage('build') {
        steps {
            lock(resource: "${env.BRANCH_NAME}") {
                node("XYZ") {
                    sh "sleep 60"
                }
            }
        }
        post {
          success {
            // do something
          }
          failure {
            // do something
          }
        }
      }
    }
}

I assume I can achieve something similar to this using try-catch inside the node block and check the status of the build in order to determine what actions to perform?

7

You can also use a node in the post block. This works

pipeline {
    agent none
    stages {
        stage("test") {
            steps {
                lock(resource: "resource") {
                  node("XYZ") {
                    sh 'echo "Hello"'
                  }
              }
           }
        }
    }
    post {
        always {
            node("infra") {
                echo "post"
            }
        }
    }
}

You would need a node in each of the post conditions though.
Certain steps don’t need an agent to run, probably sending an email as long as you don’t want to attach a file from the workspace (just guessing here haven’t tried this out).
So best to try out if you really need an agent in your post steps

1 Like