Dynamic number of stage

Using Jenkins
1

Hi,
I would like to change the number of stage and that some of them would be parallel.
Currently I have this script, the important is a the end in the “stage(‘Tests’)” with the test category 1 and category 2.

pipeline {
    agent { label 'my_agent'}
    stages {
        stage('stage_1') {
            steps {
                script {
                    sh '''
                    echo "stage_1"
                    '''
                }
            }
        }
        stage('stage_2') {
            agent { node { label 'agent_stage2' } }
                when {
                    expression {
                        return (params.PREBUILT_BIN_PATH != '');
                    }
                }
                steps {
                    dir ('client_binaries') {
                        script {
                            if (!clientBinariesExist(VERSION)) {
                                logger.info()
                                sh '''
                                echo 
                                '''
                            }
                            logger.info("stage_2")
                        }
                    }
                }
        }
        stage('Tests') {
            parallel {
                stage('tests category 1') {
                    steps {
                        dir('build_ct_m') {
                            script {
                                println("Running test cat 1")
                            }
                        }
                    }
                }
                stage('tests category 2') {
                    steps {
                        dir('build_ct_m') {
                            script {
                                println("Running test cat 2")
                            }
                        }
                    }
                }
            }
        }
    }
}

Currently test category 1 and 2 are well run in parallel. But now I would like to have a dynamic number of test (ie dynamic number of stage ‘test category X’) that will depend on a given parameter given to the job.
I tried for for loop after the parallel keyword, but groovy was complaining that he wants a stage declaration at this point.

Thank.

2

I don’t think you can do dynamic stages in declarative, only scripted. I think I am understanding what you want, but maybe you can describe it a bit more?

3

I would like to run some identical stages in parallel mode and the number of them depends on a variable (parameter given to the pipeline).
I managed to get the variable number of tests, but these ones are not started in parallel, there are lunch in sequential mode.

Here is my last code, here the stages “tests category 1” and “tests category 2” are well run in parallel. But the different iterations of each test category (ie '“Test CAT 1 iter ’ + i”) are run in sequential mode. I would like them to run in parallel mode also.

Thanks.

pipeline {
    agent { label 'my_agent'}
    stages {
        stage('stage_1') {
            steps {
                script {
                    sh '''
                    echo "stage_1"
                    '''
                }
            }
        }
        stage('stage_2') {
            agent { node { label 'agent_stage2' } }
                when {
                    expression {
                        return (params.PREBUILT_BIN_PATH != '');
                    }
                }
                steps {
                    dir ('client_binaries') {
                        script {
                            if (!clientBinariesExist(VERSION)) {
                                logger.info()
                                sh '''
                                echo 
                                '''
                            }
                            logger.info("stage_2")
                        }
                    }
                }
        }
        stage('Tests') {
            parallel {
                stage('tests category 1') {
                    steps {
                        dir('build_ct_CAT1') {
                            script {
								for(int i=1; i < params.CAT1_TESTRUN_NUMBER.toInteger() + 1; i++) {
									stage('Test CAT 1 iter ' + i) {
										println("Running test cat 1")
									}
								}
                            }
                        }
                    }
                }
                stage('tests category 2') {
                    steps {
                        dir('build_ct_CAT2') {
                            script {
								for(int j=1; j < params.CAT2_TESTRUN_NUMBER.toInteger() + 1; j++) {
									stage(''Test CAT 2 iter " + j) {
										println("Running test cat 1")
									}
								}
                            }
                        }
                    }
                }
            }
        }
    }
}
4

Declarative pipeline (ones that start with pipeline) are intended to be fully parable before the job starts.

What you are describing is not only doable but by design and documented in scripted pipelines

5

Sorry, I’m not getting it.

6

What Gavin explained is that you cannot use a Declarative Pipeline since, by design, it cannot be dynamic (everything needs to be known at the start).
But you can use a Scripted Pipeline instead. The parallel step takes a map (Pipeline: Groovy) so you can create the map dynamically and pass it to the parallel step. A quick example:

def branches = [
    'a branch': aBranch(),
    'another branch': anotherBranch()
]

parallel branches

def aBranch() {
    return {
        node {
            sh 'echo "hello from branch 1"'
        }
    }
}

def anotherBranch() {
    return {
        node {
            sh 'echo "hello from branch 2"'
        }
    }
}
7

Thanks all for you help.