Passing Metadata Among Jobs

Using Jenkins
1

Hi Jenkins experts,

Our Jenkins pipeline main job triggers other worker jobs with the build command (also we use the DSL plugin for generating all the jobs). The main job passes data to worker jobs through the parameters field in the build command. And those data controls how the worker job should be executed etc. The params will be configurable by users at the worker job’s Jenkins UI manually also.

I have some use cases, however, require passing metadata to the worker jobs so it can be used later for analysis purpose only. So ideally those metadata passed to worker jobs shouldn’t be exposed as manually configurable params. What is the best way to achieve this? Thanks in advance!

2

Hi @current817,

Wouldn’t environment variables work for you?

pipeline {
    agent any
    environment {
        MY_METADATA = 'some value'
    }
    stages {
        stage('Example') {
            steps {
                echo "The metadata is: ${env.MY_METADATA}"
            }
        }
    }
}
3

How does the main job know which params it needs to pass to the worker jobs when users can modify the jobs via the UI? And when you use JobDSL doesn’t that cause problems when user modified the job via UI and the seed job runs it might revert the user changes.

As you already use JobDSL I would forbid changing jobs via UI, then you have full control and can ensure the parameters for metadata are always there.
Assuming you have the JobDSL data in git you can work with e.g. pull requests when your users need to modify the parameters. That way you automatically also track job changes without the need to use jobconfighistory plugin

4

@mawinter69 Good questions.

When the jobs are generated (main job + worker jobs), they have a series of default params and pipelines will be run based on that. But we do expose those params to UI so the user can later trigger the jobs with their customization. (e.g. If a user needs to debug something and only want to run with that worker job config, they can tweak those params and just trigger the job). Thus we don’t want to completely disable the params exposed to UI. But there are certain metadata we want to pass to the worker jobs.

For example, in the worker job, I want to understand who is the parent job that triggered the worker. We may have other use cases that need information from the parent.

5

Thanks @poddingue. The main job may trigger multiple worker jobs, and the worker jobs don’t necessarily share the same metadata. So environment variable seems less ideal.

1 Like
6

Does that mean the users are able to configure the jobs? Or is it that they can just run the job and then choose different values for the parameters when they start the jobs. That is a big difference.

If your users are not able to modify the jobs themselves, you can use the Hidden Parameter plugin. Though it will not stop users to trigger jobs via the rest api and then modify the parameters.

For example, in the worker job, I want to understand who is the parent job that triggered the worker. We may have other use cases that need information from the parent.

The information what triggered a job is always available in a run via the Cause. There is no need to provide this information as a parameter.

7

Thanks Markus @mawinter69 . Once the jobs are generated, users can only adjust the params on UI (through edit boxes) and then start the job. Looks like the Hidden Parameter plugin does exactly what I need. Thanks!

Thanks for the idea of getting the Cause from currentBuild.rawBuild. I will give it a try. One additional question, for the Blueoccean view, do you know how does the UI gets all hierarchies of all the jobs and its worker jobs etc. Is it also through parsing the build cause?

8

Not sure what you mean with worker jobs in the context of Blue Ocean. But in general the cause of a job can also be that it is was triggered by another job, which itself was triggered by another job.

Note the Blue Ocean is in maintenance mode, means it will not get any new development only severe bugs or security issues get fixed.