How to reuse sequential stages in jenkins

Using Jenkins
1

I have few parallel stages under which there are multiple sequential stages. Only thing that differs within those stages is few environment variables. Here is how my code looks like

#!groovy
@Library(<sharedLib>) _
def GIT_COMMIT_EMAIL

pipeline {
    agent none
    parameters {
        booleanParam(name: 'uninstall_package', defaultValue: false)
        booleanParam(name: 'install_package', defaultValue: false)
        booleanParam(name: 'managed_package', defaultValue: false)
        booleanParam(name: 'compile_apex', defaultValue: false)
        booleanParam(name: 'clear_org', defaultValue: false)
        string(name: 'namespace', defaultValue: '', description: 'Package namespace')
        string(name: 'version', defaultValue: '', description: 'Package version')
        string(name: 'org', defaultValue: 'beta', description: 'org')
        string(name: 'gitbranch', defaultValue: 'master', description: 'Git Branch to build from')
        string(name: 'retry', defaultValue: '3', description: 'Number Of Retries/Reruns for failed Pytest')
        string(name: 'pw_version', defaultValue: 'latest', description: 'Playwright Version')
    }

    options {
        buildDiscarder(logRotator(artifactNumToKeepStr: '10'))
        disableConcurrentBuilds()
        timestamps()
        timeout(time: 3, unit: 'HOURS')
    }

    stages {
        stage("Installing & Running Tests Stage")
        {
            agent {
                docker {
                    args "${DOCKER_PARAMS}"
                    image <image>
                    label <label
                    registryCredentialsId <credential>
                    registryUrl <url>
                }
            }
            stages {
                // No Cleanup As Docker is run with root user and as a result generated folders/files get root ownership
                stage('Checkout SCM') {
                    steps {
                        echo "Checkout"
                    }
                }
                stage("Setup and Run Tests in Parallel"){
                    stages{
                        stage('Setup CCI for Org') {
                            steps{
                                setup_cci_org()
                            }
                        }
                        stage("Create Scratch org and Run tests"){
                            parallel {
                                stage("Running against Auto_org_1"){
                                    environment{
                                        auto_org_name = "auto_org_1"
                                        auto_org_kexpr = "${params.kexpr1}"
                                    }
                                    stages{
                                        stage('Create Scratch Org') {
                                            steps{
                                                create_scratch_org("${org}")
                                            }
                                        }
                                        stage('Install Managed Package') {
                                            when {
                                                expression { (params.install_package == true) && (params.managed_package == true) }
                                            }
                                            steps {
                                                install_package_for_org("${org}",  params.version, params.namespace)
                                            }
                                        }

                                        stage('Copy Config & Decrypt Secrets') {
                                            steps{
                                                echo "Copy Config & Decrypt Secrets"
                                            }
                                        }
                                    }
                                }
                                stage("Running against Auto_org_2"){
                                    environment{
                                        auto_org_name = "auto_org_2"
                                        auto_org_kexpr = "${params.kexpr2}"
                                    }
                                    steps{
                                        echo "Running against Auto_org_2"
                                        // Repeat stages from Auto_org_1
                                    }
                                }
                            }
                        }
                    }
                }
            }
            post {
                always {
                    echo 'Post Execution'
                    script{remove_scratch_org("auto_org_1")}
                    script{remove_scratch_org("auto_org_2")}
                }
                failure {
                    echo "Tinkering Failed"
                }
            }
        }
    }
}
def create_scratch_org(org)
{
    echo "create_scratch_org ${org}"
}
def remove_scratch_org(org)
{
    echo "remove_scratch_org ${org}"
}
def setup_cci_org()
{
    echo "Setting up cci org"
}
def install_package_for_org(org, version, namespace)
{
    echo "install_package_for_org(org, version, namespace)"
}

What I would like to do is capture all the stages under stage("Running against Auto_org_1") and put it in a method. I tried doing this by changing my code as follows –

#!groovy
@Library(<sharedLib>) _
def GIT_COMMIT_EMAIL

pipeline {
    agent none
    parameters {
        ...
    }

    options {
        ...
    }

    stages {
        stage("Installing & Running Tests Stage")
        {
            agent {
                docker {
                    ...
                }
            }
            stages {
                stage('Checkout SCM') {
                    steps {
                        echo "Checkout"
                    }
                }
                stage("Setup and Run Tests in Parallel"){
                    stages{
                        stage('Setup CCI for Org') {
                            steps{
                                setup_cci_org()
                            }
                        }
                        stage("Create Scratch org and Run tests"){
                            parallel {
                                stage("Running against Auto_org_1"){
                                    environment{
                                        auto_org_name = "auto_org_1"
                                        auto_org_kexpr = "${params.kexpr1}"
                                    }
                                    steps{
                                        script{
                                            all_steps("${auto_org_name}")
                                        }
                                    }
                                }
                                stage("Running against Auto_org_2"){
                                    environment{
                                        auto_org_name = "auto_org_2"
                                        auto_org_kexpr = "${params.kexpr2}"
                                    }
                                    steps{
                                        echo "Running against Auto_org_2"
                                        // Repeat stages from Auto_org_1
                                    }
                                }
                            }
                        }
                    }
                }
            }
            post {
                always {
                    echo 'Post Execution'
                    script{remove_scratch_org("auto_org_1")}
                    script{remove_scratch_org("auto_org_2")}
                }
                failure {
                    echo "Tinkering Failed"
                }
            }
        }
    }
}

def all_steps(org)
{
    stage("All Steps stage"){
        stages{
            stage('Create Scratch Org') {
                steps{
                    create_scratch_org("${org}")
                }
            }
            stage('Install Managed Package') {
                when {
                    expression { (params.install_package == true) && (params.managed_package == true) }
                }
                steps {
                    install_package_for_org("${org}",  params.version, params.namespace)
                }
            }
            stage('Copy Config & Decrypt Secrets') {
                steps{
                    echo "Copy Config & Decrypt Secrets"
                }
            }
        }
    }
}

def create_scratch_org(org)
{
    echo "create_scratch_org ${org}"
}
def remove_scratch_org(org)
{
    echo "remove_scratch_org ${org}"
}
def setup_cci_org()
{
    echo "Setting up cci org"
}
def install_package_for_org(org, version, namespace)
{
    echo "install_package_for_org(org, version, namespace)"
}

But this throws an error java.lang.NoSuchMethodError: No such DSL method 'stages' found among steps

I have now spend enough time trying to figure out this, now hoping experts over here can guide me further :slight_smile:

I know there is option to also move the code to shared library, but before that I would like this to work as a separate method. I assume if I put the body of method under vars folder and wrap it in a call method same thing can be called as a script inside pipeline. Which is why local method is first thing I want to try as its faster to validate changes that way.

2

So I’m a little distracted by work, so only had a chance to skim not fully follow the super well detailed question. But if i’m reading this right, your wanting to make a set of stages that can be re-used over and over again in your jenkins file?

So as I understand it, the big difference between declarative and scripted pipelines is that declarative has all the flow control pre-parsed, while scripted is entirely evaluated as it goes. So that means you can’t really add new stages at runtime.

The solutions I see:

  1. Switch your pipeline to scripted. Then you can add stages and sub stages, and as much as you want, you can use loops, and functions, etc. You’ll loose access to some of the flow control ones like when, but you’ll get access to raw if checks.
  2. Treat the jenkinsfile as the output of another script. echo part1()\npart2()\npart3() > Jenkinsfile type thing, and have a job that runs that generates the jenkinsfile once in a while
  3. Don’t split them up into seperate stages, just have the steps as a function
  4. Call a new job via build with parameters for those parallel steps

I think 3 is the easiest, with maybe 1 being the best for your sanity long term

Hope that helps.

3

Hi,

Thanks for the reply.
Yes that is exactly what I want to do

Hmm, I was hoping to not go to scripted version, as DSL gives me less but easily readable code.
So true 1 and 3 might work
3 would reduce pipeline visibility in blueocean view
2 I didn’t quiet understand how that is done.
4 never done that, but I think it is an overkill difficult for a new person to take over job if am not around.

4

yea, I don’t like builds triggering builds, but it is a technique some people use for reusable code.

For 2 I was just thinking of making a script called generate_pipeline.sh or something

echo "pipeline {"
echo "  stages { stage('first stage') { steps { echo hi } } }"
echo "}"

then /system/generate_pipelines.sh > Jenkinsfile

That way you don’t have to have complex dynamic steps in your pipeline, but you do have the declarative pipeline committed, and then when you want to make changes to the pipeline, just re-run the script.

Not saying its a good idea, just another option.

Make sure you check out the great work timja has been doing on Blue Ocean but in classic UI - #16 by timja It doesn’t fix your problem, but as blueocean really isn’t getting updates its good to know about the alternatives