Wrapper classes for JobDSL job definitions - scalable job definition code

Using Jenkins Ask a question
1

I have been trying to figure out this problem for a long time now. I thought I had found a solution, but it looks like the world has moved on from it.

Some background

First, a bit of background. I have used the JobDSL plugin to provision Jenkins jobs using IaC principles. My jobs definition section would be configured to point to a jenkins-scripts repo that contained the pipeline scripts to run for certain jobs (some jobs would run the same scripts with different input values). And my scripts would import a jenkins-shared-library to import shared functions for things like “compile Java code”, “unit test Java code”, “build container image”, “trivy scan container image”, “push container image”, etc.

I am quite happy with my scripts repo and shared-library repo as they allowed me to follow DRY and KISS principles, but I was not able to do the same with my job definitions written in JobDSL syntax.

Project repo structures & code examples

Here are some code example to better demonstrate the structure I am used to:

Here’s the project structure. Just consider all project and app directories to have the same structure, so I only show the 1st one.

jenkins-jobs/
├── project1/
│   ├── directory.groovy
│   ├── app1/
│   │   ├── directory.groovy
│   │   ├── automation_testing.groovy
│   │   ├── build/
│   │   │   ├── directory.groovy
│   │   │   ├── java_services.groovy
│   │   │   ├── node_services.groovy
│   │   │   └── python_services.groovy
│   │   ├── deploy/
│   │   │   ├── directory.groovy
│   │   │   ├── deploy_services.groovy
│   │   │   └── deploy_all_services.groovy
│   │   └── release/
│   │       └── directory.groovy
│   └── app2/
│       ├── build
│       ├── deploy
│       └── release
└── project2/
    ├── app1/
    │   ├── build
    │   ├── deploy
    │   └── release
    └── app2/
        ├── build
        ├── deploy
        └── release

Here are some examples of the directory.groovy files. These are done in a way so that the repo structure appears the same as in the Jenkins webUI:

# project1/directory.groovy
String F_ROOT = 'PROJECT-1'

folder(F_ROOT) { 
    configure {
           # I add some permissions config here to ensure teams only see folders related to them
        }
    }
}

# project1/app1/directory.groovy
String F_ROOT = 'PROJECT-1/APP-1'

folder(F_ROOT) {}

# project1/app1/build/directory.groovy
String F_ROOT = 'PROJECT-1/APP-1/BUILD'

folder(F_ROOT) {}

# project1/app1/deploy/directory.groovy
String F_ROOT = 'PROJECT-1/APP-1/DEPLOY'

folder(F_ROOT) {}

And here is an example of the project1/app1/build/java_services.groovy:

String F_ROOT = 'PROJECT-1/APP-1/BUILD'
String JOB_NAME = 'job_build'

// Execute this task at a random minute within the 11 PM hour, every day
// Execute this task at a random minute within the 11 PM hour, every Sunday
def CRON_SCHEDULE_EVERY_DAY = 'H 23 * * 1,2,3,4,5,6,0'
def CRON_SCHEDULE_EVERY_SUNDAY = 'H 23 * * 0'

def SERVICES = [
    "service1-name",
    "service2-name",
    "service3-name"
]

// TODO: Try to use this list in the activeChoiceReactiveParam
def ENVIRONMENTS = [
    'ALPHA',
    'STAGING',
    'AUTOMATION',
    'DEV',
    'TEST1',
    'TEST2'
]


for (SVC in SERVICES) {

    SVC_NAME = SVC.replaceAll('-', ' ')
    SVC_PATH = SVC.replaceAll('-', '_')

    pipelineJob("${F_ROOT}/${JOB_NAME}_${SVC_PATH}") {

        displayName(SVC_NAME.toUpperCase())

        parameters {
            gitParameter {
                name('SERVICE_BRANCH')
                type('PT_BRANCH')
                branch('')
                defaultValue('')
                description('Please select the branch to build the service')
                tagFilter('')
                sortMode('DESCENDING_SMART')
                selectedValue('TOP')
                useRepository(SVC)
                quickFilterEnabled(true)
            }
            gitParameter {
                name('BASE_IMAGE_BRANCH')
                type('PT_BRANCH')
                branch('')
                defaultValue('main')
                description('Please select the branch for the base image')
                tagFilter('')
                sortMode('DESCENDING_SMART')
                selectedValue('TOP')
                useRepository('docker-images')
                quickFilterEnabled(true)
            }
            activeChoiceParam('DEPLOY_AFTER_BUILD') {
                description('Do you want to deploy to environment after a successful build?')
                filterable(false)
                choiceType('SINGLE_SELECT')
                groovyScript {
                    script('return ["NO:selected", "YES"]')
                    fallbackScript('"The choice script failed"')
                }
            }
            activeChoiceReactiveParam('ENVIRONMENT') {
                description('Select an environment to deploy to')
                filterable(false)
                choiceType('SINGLE_SELECT')
                groovyScript {
                    // TODO: Find a better way to load this list into the script (try to use ENVIRONMENTS ArrayList)
                    script("if (DEPLOY_AFTER_BUILD.equals('YES')) { return ['ALPHA', 'STAGING', 'DEV', 'AUTOMATION', 'TEST1', 'TEST2'] }")
                    fallbackScript('"The choice script failed"')
                }
                referencedParameter('DEPLOY_AFTER_BUILD')
            }
            hidden {
                name('SERVICE_NAME')
                defaultValue(SVC)
                description('The name of the service being built')
            }
        }

        environmentVariables {
            env('FAILED_STAGE', 'Pipeline start')
            keepBuildVariables(true)
            keepSystemVariables(true)
            overrideBuildParameters(false)
        }

        properties {
            pipelineTriggers {
                triggers {
                    parameterizedTimerTrigger {
                        parameterizedSpecification("""
                            ${CRON_SCHEDULE_EVERY_DAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                            ${CRON_SCHEDULE_EVERY_DAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                            ${CRON_SCHEDULE_EVERY_DAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                            ${CRON_SCHEDULE_EVERY_SUNDAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                            ${CRON_SCHEDULE_EVERY_SUNDAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                            ${CRON_SCHEDULE_EVERY_SUNDAY} %SERVICE_BRANCH=main;BASE_IMAGE_BRANCH=main;DEPLOY_AFTER_BUILD=NO;
                        """)
                    }
                }
            }
        }

        logRotator {
            numToKeep(30)
        }
        
        definition {
            cpsScm {
                scm {
                    git {
                        remote {
                            url('ssh://git@${my-git-domain}/jenkins-scripts.git') # this is the repo where all the scripts are stored
                        }
                        remote {
                            url('ssh://git@${my-git-domain}/docker-images.git')
                        }
                        remote {
                            url("ssh://git@${my-git-domain}/${SVC}.git")
                        }

                        branch('main')
                        extensions {
                            cleanBeforeCheckout()
                        }
                    }
                }
                scriptPath('build/Jenkinsfile_build_java_service.groovy') # this is the path for the script to run, which is found in the `jenkins-scripts` repo.
                                                                          # all my scripts utilise a jenkins-shared-library to follow the DRY principle.
            }
        }
    }
}

So you can see that I have some repeatability using the for to iterate over the SERVICES list, which is fine and I have 1 script for 1 type of job/service (build/java, deploy/java, build/python, etc.), but I want more re-usability.

The primary issue

My top problem here is that for any new project, or new app within a project, I have to copy/paste a file and then edit it according to what is needed, for example, the SERVICES list or the ENVIRONMENTS list.
There are usually some other larger changes as well like different
Cron schedules, potentially checking out code from a different git repository domain, etc.

I don’t want to copy/paste and then edit the new files in this way as it just seems silly, wrong and while it is scalable to some extent, I would like it to be even easier, and more scalable.

A possible solution

A came across some amazing articles by Marc Esher, here is a link to the 1st article. There are 4 or 5 different articles on Marc’s website detailing how he used custom groovy classes as a wrapper around JobDSL syntax to create re-usable job definition configurations. Here is a link to 1 such article that best describes the entire workflow of creating a re-usable JobBuilder class and the using it in a groovy script that is read and used by the JobDSL Seed job.

As soon as I saw this, I was jumping for joy! It was exactly the sort of thing I needed. It wrapped up job configurations very nicely, while still allowing some customisation when calling the JobBuilder class in case it wasn’t quite what you needed.

I immediately set about trying to make a simple repository to prove that this works.

A new problem

Unfortunately, the articles I have linked are almost 10 years old and in that time there have been many updates to Jenkins and it’s plugin ecosystem.

The main issue I came across was JobDSL’s integration with the Script Security plugin (some info about that in a wiki page here).
This essentially blocks the use of custom classes within script execution, which directly breaks this method of wrapping JobDSL syntax in custom Groovy classes.
The only way that I found enabled the use of custom Groovy classes was disabling JobDSL script security entirely, which is more than undesirable!
I also had some annoying issues with the JobDSL steps AdditionalClasspath setting, but that is minor compared to disabling script security globally.

A call for help

I’ve been trying a bunch of different things for about 2-3 months trying to get this sort of thing to work, as it is such a great solution, but I can’t quite find how to get it working without globally disabling JobDSL script security, which I definitely do not want to do.

I would be ok with disabling script security for only the JobDSL seed job, as that will only be accessed and ran by Jenkins administrator users, but I don’t think it is possible to disable script security for 1 single job.

I have also seen that the Script Console can be used with no restrictions, so this could be another option as that would only be accessed by administrator users as well, but I’d much prefer to use the JobDSL Seed job as it’s fairly easy to understand and it is pre-configured to run all the correct files, so all a Jenkins dev needs to do is update the branch it looks at and run the job.

Any help on this, even just suggestions and nudges in the right direction would be greatly appreciated.

Thank you for taking the time to read this rather long post!

2

This blog post might give you some peculiar ideas: Getting Started With Jenkins Job DSL Plugin for Standardising Your Pipelines · Jamie Tanna | Software Engineer . The core idea is that you’d have an actual Java/Groovy library (in form of a .jar) with helpers for your JobDSL, and then you’d run something like

jobDsl(
  targets: 'myJobs/**/*.groovy',
  additionalClasspath: 'myJDSLwrapper/build/libs/*.jar',
)

The main issue that I see is that you would have a hard time befriending this with JCASC (which allows for jobs: - file: "${CASC_JENKINS_CONFIG}/jobs/myjob-dsl.groovy" ... right in its YAML. The seed job should work fine though.

Additionally, Testing DSL Scripts · jenkinsci/job-dsl-plugin Wiki · GitHub , while somewhat dated, might also give you an idea or two.

3

Thank you for your reply @Artalus.

I’ve had a quick look at the links you provided and I think the “Jamie Tanner” website link shows essentially the same thing I have highlighted, but it’s been done ever so slightly differently.

The main issue I’ve come across is that JobDSL Script Security must be globally disabled to use custom groovy classes.
In the JobDSL seed job config, AdditionalClasspath could be set with Script Security enabled in previous versions of Jenkins/JobDSL + Script Security plugins. On the latest versions, enabling script security removes the option entirely. However, custom groovy classes are not whitelisted by the Script Security plugin, and so it won’t allow them to be used, even when AdditionalClasspath is set.

The link you provided shows very similar wrapper classes that wrap JobDSL syntax. The blog post states " (Note that Script Security on Jenkins may cause you issues - please refer to the Job DSL Script Security documentation for more info if you’re seeing compilation errors )".
This method has the exact same issue as I have described already. Script security would need to be disabled globally (or just for the seed job, if thats even possible) for these classes to be usable.
I’m not entirely against disabling script security for the seed job, but I definitely don’t want to disable it globally.

I don’t think there are any problems with getting the JobDSL seed job to be created by JCasC, I have that in my current setup and it works just fine. Although, you might be referring to how the seed job is done in the blog post you linked, that I can’t comment on.

Thank you for adding the link to the JobDSL “Testing DSL Scripts” page!
I had come across some other testing resources and that was another reason I wanted to try using these wrapper classes, as with my current setup, I don’t think there is a good way to test it other than manually (which I have absolutely hated) and using Groovy classes would allow the use of automation testing frameworks like Spock.

Thank you for taking the time to read my long post and posting a reply.
It is unfortunate that I don’t believe it solves the main issue of Script Security getting in the way of things.