Give Python script access to pipeline steps

Using Jenkins Ask a question
,
1

Does anyone have a way to access Jenkins pipeline steps from an external script? We’re trying to centralize as much of our jobs as possible in Python scripts, but the problem is the inability to access steps from the script. For example, to archive artifacts or publish documentation, the script needs to communicate that information back to Groovy so the Groovy can call archiveArtifacts, etc.

One thing I badly miss in Python is the ability to create stages to organize which part of the job is running; now that it’s in Python the output is all under a single sh python block.

I’m fairly sure there’s no built-in way to do this, so I’m hoping for either a plugin or some clever Groovy scripting that makes it possible. I’ve tried to do it via either opening a FIFO or a socket from the Groovy script that the Python script can use to talk to it, but ran into a lot of complexity trying to make that work.

2

I do not think this is a realistic expectation - the Java/Groovy code runs on the Controller, Python would run on Agent/Node. My suggestion would be to turn this insideout - create your components and have Groovy Pipeline call them - if you want to reuse the same components without Groovy you would have to re-create the outer script that calls them but all the steps would be the same.

Generally you should not things needed for build vs things needed for orchestration - put your build inside whatever is the build tool of your choosing, then only run the orchestration parts you need Jenkins for (staging environment, handling output an reports, etc) in Jenkins, and where the build would be - call your build tool. This way you only get Jenkins specific code in Jenkins, and your devs can build things locally without worrying about Jenkins or any other CICD tool

HTH

3

Any of the alternative runtimes have been removed, I can find the blog post that go into detail, but they were really old, nobody was maintaining them, and making it hard to upgrade. So now its java only again.

I recommend just using pipeline, as @mlasevich hints at, its pretty simple to do basic flow there (steps, stages, etc).

If you want to write everything in python, you can see two “hacks”

  1. Have a bunch of jobs that do things, and call them as parameters. So job/uploadArtifact/buildWithParameters?fileContents=base64contents
  2. Have python write out a Jenkinsfile.python, then your basic jenkinsfile just calls load:./Jenkinsfile.python

But yea, I wouldn’t recommend either of them. Basic groovy shouldn’t be hard to learn, and your pipeline can just call your python scripts to do any real work.

4

We have a monorepo with multiple languages and in order to save resources we have a ‘prepare’ stage that looks at the list of changed files in a PR and based on that decides which stages actually need to run.

The logic is written in python, which writes to a property file which the declarative pipeline then reads.

import groovy.transform.Field
@Field Map pr_focus = [:]
pipeline {
    ...
    stages {
        stage ('Prepare') {
            agent { ... }
            }
            steps {
                container('python') { sh "jenkins/pr_focus.py > pr_focus.txt" }
                script { pr_focus = readProperties(file: 'pr_focus.txt') }
            }
        }
        stage('Tests') {
            parallel {

                stage('Check Foo') {
                    when {
                        not { equals expected: '1', actual: pr_focus.SKIP_FOO_STAGES }
                        beforeAgent true
                    }

I hope this helps.