httpRequest with two credentials - how to avoid insecure interpolation of sensitive variables?

blueice_haller · March 15, 2023, 11:10am

Hello,
I would like to understand how my code need to look like, so that I don’t see the warning:

The following steps that have been detected may have insecure interpolation of sensitive variables (click here for an explanation):
httpRequest: [BACKUP_PASSWORD]

The API of an non public application requests, that I authenticate against the API with Username and Password, stored in BACKUP_CREDENTIALS and provide an password (Secret BACKUP_PASSWORD) in JSON format in the request body.
When restoring the backup zip file, both credentials need to be provided again.

import java.time.*
import java.time.format.DateTimeFormatter

def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd_hhmmss',TimeZone.getTimeZone('Europe/Berlin'))
println timeStamp

pipeline {
	agent any
	
	stages {
		// https://www.jenkins.io/doc/book/pipeline/syntax/#supported-credentials-type
		stage('Backup') {
			environment {
				HOSTNAME = "example.com"
				BACKUP_PASSWORD = credentials('RESTORE')
			}
			steps {
				timeout(time: 120, unit: 'SECONDS') {
					script {
						// https://www.jenkins.io/doc/pipeline/steps/http_request/
						// https://plugins.jenkins.io/http_request/
						
		                withCredentials([string(credentialsId: 'RESTORE', variable: 'BACKUP_PASSWORD')]) {
		                    def response = httpRequest url: 'https://' + HOSTNAME + ':1234/api/backup',
		                    acceptType: 'APPLICATION_ZIP',
		                    authentication: 'BACKUP_CREDENTIALS',
		                    consoleLogResponseBody: false,
		                    contentType: 'APPLICATION_JSON',
		                    httpMode: 'POST',
		                    ignoreSslErrors: true,
		                    outputFile: BUILD_NUMBER + '_backup_' + timeStamp + '_' + HOSTNAME + '.zip',
		                    requestBody: """
								{"password": '$BACKUP_PASSWORD'}
							""",
		                    timeout: 120,
		                    wrapAsMultipart: false
		                }
					}
				}
			}
		}
	}
}

Thank you in advance.

poddingue · March 17, 2023, 9:01am

Hello @blueice_haller and welcome to this community

The warning is indicating that there may be sensitive data being interpolated in the requestBody field of the httpRequest step. In your case, the BACKUP_PASSWORD variable is being used directly in the JSON request body, which could be insecure if the value of BACKUP_PASSWORD is sensitive.

To avoid this warning and make your code more secure, you can use the withCredentials step to securely pass the BACKUP_PASSWORD value to the httpRequest step. Here’s an example of how to modify your code:

import java.time.*
import java.time.format.DateTimeFormatter

def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd_hhmmss',TimeZone.getTimeZone('Europe/Berlin'))
println timeStamp

pipeline {
    agent any
    
    stages {
        // https://www.jenkins.io/doc/book/pipeline/syntax/#supported-credentials-type
        stage('Backup') {
            environment {
                HOSTNAME = "example.com"
                BACKUP_PASSWORD = credentials('RESTORE')
            }
            steps {
                timeout(time: 120, unit: 'SECONDS') {
                    script {
                        // https://www.jenkins.io/doc/pipeline/steps/http_request/
                        // https://plugins.jenkins.io/http_request/
                        
                        withCredentials([string(credentialsId: 'RESTORE', variable: 'BACKUP_PASSWORD')]) {
                            def requestBodyJson = [:]
                            requestBodyJson['password'] = "${BACKUP_PASSWORD}"
                            def requestBody = JsonOutput.toJson(requestBodyJson)
                            def response = httpRequest url: 'https://' + HOSTNAME + ':1234/api/backup',
                            acceptType: 'APPLICATION_ZIP',
                            authentication: 'BACKUP_CREDENTIALS',
                            consoleLogResponseBody: false,
                            contentType: 'APPLICATION_JSON',
                            httpMode: 'POST',
                            ignoreSslErrors: true,
                            outputFile: BUILD_NUMBER + '_backup_' + timeStamp + '_' + HOSTNAME + '.zip',
                            requestBody: requestBody,
                            timeout: 120,
                            wrapAsMultipart: false
                        }
                    }
                }
            }
        }
    }
}

In this modified code, the BACKUP_PASSWORD value is passed to the httpRequest step using the withCredentials step, which securely handles the sensitive value. The JSON request body is constructed using a map and JsonOutput.toJson method, rather than directly interpolating the BACKUP_PASSWORD variable. This approach should prevent the warning message from being displayed.

blueice_haller · March 17, 2023, 4:02pm

Thank you very much. It works.

I need to import groovy.json.* to not get this Error:

groovy.lang.MissingPropertyException: No such property: JsonOutput for class: groovy.lang.Binding

poddingue · March 17, 2023, 4:59pm

Thanks a lot for your feedback, always happy to help.

blueice_haller · March 17, 2023, 10:50pm

Should I create an additional thread / question?

I’m trying to model this with Blue Ocean.
timeout is named “Enforce time limit”.
httpRequest is named “Perform an HTTP Request and return a response object”

How to model script and withCredentials?
Do I need to use “Evaluate a Groovy source file into the Pipeline script” or “Run arbitrary Pipeline script”?

I found out the answer by copy the code including imports from one to another Jenkinsfile.
script is named “Run arbitrary Pipeline script”.
The copy the code from withCredentials into the field.

blueice_haller · March 17, 2023, 11:17pm

Should I create an additional thread / question?

How I can run my code on e.g. 10 target HOSTNAMEs?

I thought that using variables in the environment and parameters is part of the solution?

Is it possible to store them in the GitHub / Workspace location in a separate file next to Jenkinsfile
and the read and process this file?

I will have a look on Pipeline Examples (jenkins.io).

poddingue · March 18, 2023, 5:41pm

When using Blue Ocean, you can use the “Run arbitrary Pipeline script” step to run arbitrary Groovy code within your pipeline. This allows you to use any Groovy code, including the withCredentials block, within your pipeline.

You can also use the “Evaluate a Groovy source file into the Pipeline script” step to load Groovy code from an external file into your pipeline. This can be useful if you have a large amount of shared or reusable code that you want to keep in a separate file.

Both of these steps can be used to include the withCredentials block in your pipeline. Just make sure to import the necessary libraries and use the correct syntax when using these steps.

poddingue · March 18, 2023, 5:45pm

Yes, using variables in the environment and parameters is definitely part of the solution to run your code on multiple target hostnames. Here’s an example of how you couldstore the hostnames in a separate file and read and process the file in your Jenkinsfile:

Create a file named hostnames.txt in your GitHub or workspace location with the list of hostnames, one per line:

hostname1
hostname2
hostname3
...

Add a parameters block to your pipeline to allow the user to input the location of the hostnames.txt file:

pipeline {
    parameters {
        string(name: 'HOSTNAMES_FILE', defaultValue: 'hostnames.txt', description: 'Location of the file containing the list of target hostnames')
    }
    ...

In the environment block, read the contents of the hostnames.txt file into a variable:

environment {
    HOSTNAMES = sh(script: "cat ${params.HOSTNAMES_FILE}", returnStdout: true).trim().split('\n')
    BACKUP_PASSWORD = credentials('RESTORE')
}

In the steps block, loop over the HOSTNAMES variable and perform the backup operation on each hostname:

steps {
    timeout(time: 120, unit: 'SECONDS') {
        script {
            for (hostname in env.HOSTNAMES) {
                def response = httpRequest url: 'https://' + hostname + ':1234/api/backup',
                acceptType: 'APPLICATION_ZIP',
                authentication: 'BACKUP_CREDENTIALS',
                consoleLogResponseBody: false,
                contentType: 'APPLICATION_JSON',
                httpMode: 'POST',
                ignoreSslErrors: true,
                outputFile: BUILD_NUMBER + '_backup_' + timeStamp + '_' + hostname + '.zip',
                requestBody: """
                    {"password": '$BACKUP_PASSWORD'}
                """,
                timeout: 120,
                wrapAsMultipart: false
            }
        }
    }
}

This will perform the backup operation on each hostname listed in the hostnames.txt file. Note that you can modify the HOSTNAMES_FILE parameter to point to a different file if needed.

blueice_haller · April 3, 2023, 2:55pm

When I merge #1, #2 + #3 + #8 together my example code looks the following - is this correct?

import java.time.*
import java.time.format.DateTimeFormatter

def timeStamp = Calendar.getInstance().getTime().format('YYYYMMdd_hhmmss',TimeZone.getTimeZone('Europe/Berlin'))
println timeStamp

pipeline {
    agent any
    
    parameters {
        string(name: 'HOSTNAMES_FILE', defaultValue: 'hostnames.txt', description: 'Location of the file containing the list of target hostnames')
    }
    
    stages {
        // https://www.jenkins.io/doc/book/pipeline/syntax/#supported-credentials-type
        stage('Backup') {
            environment {
                HOSTNAME = "example.com"
                HOSTNAMES = sh(script: "cat ${params.HOSTNAMES_FILE}", returnStdout: true).trim().split('\n')
                BACKUP_PASSWORD = credentials('RESTORE')
            }
            steps {
                timeout(time: 120, unit: 'SECONDS') {
                    script {
                        for (hostname in env.HOSTNAMES) {
                            // https://www.jenkins.io/doc/pipeline/steps/http_request/
                            // https://plugins.jenkins.io/http_request/
                            
                            withCredentials([string(credentialsId: 'RESTORE', variable: 'BACKUP_PASSWORD')]) {
                                def requestBodyJson = [:]
                                requestBodyJson['password'] = "${BACKUP_PASSWORD}"
                                def requestBody = JsonOutput.toJson(requestBodyJson)
                                // def response = httpRequest url: 'https://' + HOSTNAME + ':1234/api/backup',
                                def response = httpRequest url: 'https://' + hostname + ':1234/api/backup',
                                acceptType: 'APPLICATION_ZIP',
                                authentication: 'BACKUP_CREDENTIALS',
                                consoleLogResponseBody: false,
                                contentType: 'APPLICATION_JSON',
                                httpMode: 'POST',
                                ignoreSslErrors: true,
                                outputFile: BUILD_NUMBER + '_backup_' + timeStamp + '_' + HOSTNAME + '.zip',
                                requestBody: requestBody,
                                timeout: 120,
                                wrapAsMultipart: false
                            }
                        }
                    }
                }
            }
        }
    }
}

I don’t understand, why hostname is not resolved and why the Pipeline fails?

+ cat hostnames.txt doesn’t show the content in the build console
URL: https://[:1234/api/backup doesn’t contain the hostname
It throws one exception and it’s not easy to understand in which line the cause is

...
[Pipeline] Start of Pipeline
[Pipeline] echo
20230403_053429
[Pipeline] node
Running on Jenkins in /jenkinsdata/example/main/workspace
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
...
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Backup)
[Pipeline] withCredentials
Masking supported pattern matches of $BACKUP_PASSWORD
[Pipeline] {
[Pipeline] sh
+ cat hostnames.txt
[Pipeline] withEnv
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 2 min 0 sec
[Pipeline] {
[Pipeline] script
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $BACKUP_PASSWORD
[Pipeline] {
[Pipeline] }
[Pipeline] httpRequest
HttpMethod: POST
URL: https://[:1234/api/backup
Content-Type: application/json
Accept: application/zip
[Pipeline] }
[Pipeline] }
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
an exception which occurred:
	in field com.cloudbees.groovy.cps.impl.ForInLoopBlock$ContinuationImpl.itr
	in object com.cloudbees.groovy.cps.impl.ForInLoopBlock$ContinuationImpl@7c90a4cf
	in field com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.target
	in object com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl@3203f900
	in field com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv.continue_
	in object com.cloudbees.groovy.cps.impl.LoopBlockScopeEnv@71263409
	in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent
	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@6df14628
	in field com.cloudbees.groovy.cps.impl.ProxyEnv.parent
	in object com.cloudbees.groovy.cps.impl.BlockScopeEnv@269571d3
	in field com.cloudbees.groovy.cps.impl.CpsClosureDef.capture
	in object com.cloudbees.groovy.cps.impl.CpsClosureDef@1bc342e4
	in field com.cloudbees.groovy.cps.impl.CpsClosure.def
	in object org.jenkinsci.plugins.workflow.cps.CpsClosure2@502d0610
	in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.closures
	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@72c07282
	in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@72c07282
Caused: java.io.NotSerializableException: java.util.ArrayList$Itr
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:274)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:341)
	at java.base/java.util.HashMap.internalWriteEntries(HashMap.java:1858)
	at java.base/java.util.HashMap.writeObject(HashMap.java:1412)
	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:89)
	at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:199)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1089)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:116)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.lambda$writeObject$1(RiverWriter.java:144)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:331)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:143)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:577)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:554)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:537)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:461)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

GitHub has been notified of this commit’s build result

Finished: FAILURE

halkeye · April 19, 2023, 12:25am

You don’t have parameters defined.

Also it means your don’t didn’t work right so it thinks it’s ] in the contents somewhere.

Why not use readJSOM or readFile instead of shhellingng it.

Topic		Replies	Views
Using safely Credentials in shell commands? Ask a question	3	2870	December 27, 2023
Interpolating sensitive values in jenkins pipeline Ask a question	2	148	March 18, 2024
Masking local strings in Groovy Using Jenkins	4	1138	May 4, 2024
Jenkins Credential Interpolation issue within shell script Ask a question question	1	966	July 15, 2023
Pipeline using curl and issues with interploation of credentials Using Jenkins	4	21742	April 28, 2022

httpRequest with two credentials - how to avoid insecure interpolation of sensitive variables?

Related topics