PATH variable is overwritten by jnlp container

mohanisch · October 7, 2022, 8:43pm

Hello,

We are using Jenkins with kubernetes plugin.
In Jenkins, various agents are set up, all with a custom image. In one image, for example, golang is installed and the PATH is adjusted accordingly:

ENV GOPATH /home/jenkins/go
ENV PATH $PATH:/usr/local/go/bin:$GOPATH/bin

Now we have noticed that in the pipeline $PATH from the golang-image is overwritten with the $PATH from the jnlp container.

Pipeline:

pipeline {
    agent {
        kubernetes {
		    inheritFrom 'golang'
		    defaultContainer 'go'
        }
    }

    stages {
        stage ('test'){
            steps {
                sh "echo ${PATH}"
                sh "echo ${env.PATH}"
                println Path
                sh "printenv"
            }
        }
    }
}

Result from job log:

+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] echo
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ printenv
....
PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
....

Similar issue we have with a java agent. We installed two java-versions and set im Dockerfile:

ENV JAVA_HOME_18 /usr/lib/jvm/jdk-18
ENV JAVA_HOME_19 /usr/lib/jvm/jdk-19

In pipeline:

sh "echo $env.JAVA_HOME_19"

has no result.

What is wrong?

Thanks in advance!
Marco

mohanisch · October 9, 2022, 1:35pm

Okay, I got it now.
We are migrating from ssh-agents to kubernetes agents.
Before, we were able to use each environment variable from agent (backed into image).

How can we still use the environment of the agents?

saper · October 9, 2022, 2:20pm

What you could try is to run two containers in one pod, jnlp-agent and your golang stuff separately. jnlp-agent is pretty cut down and meant only to run the agent itself.

mohanisch · October 24, 2022, 9:32am

@saper sorry for late reply.

The jnlp container is created automatically by kubernetes plugin.

We are facing more and more issues… most of them are related to PATH.

While running the pipeline we cannot add something to PATH like

PATH=PATH:/path/to/some/bin

This is for example added es ENV in the Dockerfile but this path is not populated.

Echo on PATH has a different output as printev for example.

What is wrong?

mohanisch · October 24, 2022, 10:02am

Here one example. $PATH is configured as environment variable in pod.

Pipeline:

pipeline {
    agent {
        kubernetes {
  		    inheritFrom 'agent-build'
  		    defaultContainer 'build'
        }
    }
    stages {
        stage('TEST') {
            steps {
                echo "Hello world"
                echo "-------------"
                sh 'echo $PATH'
                echo "-------------"
                sh "PATH=$PATH:/some/path/to/bin"
                sh "echo $PATH"
                echo "-------------"
                sh """
                    PATH=$PATH:/some/path/to/bin
                    echo $PATH
                """
            }
        }
    }
}

Output:

[Pipeline] stage
[Pipeline] { (TEST)
[Pipeline] echo
Hello world
[Pipeline] echo
-------------
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

mohanisch · August 2, 2023, 7:40am

This issue is still existing… Here is the print how the pod got created:

apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins-test.jenkins.svc.cluster.local:8080/job/TEST/job/test-go/25/"
    runUrl: "job/TEST/job/test-go/25/"
  labels:
    jenkins/jenkins-test-jenkins-agent: "true"
    jenkins/label-digest: "bf10494b3f401f3387218f696f5b0565e9ca22cd"
    jenkins/label: "TEST_test-go_25-07z28"
  name: "test-test-go-25-07z28-kbtj7-hjvxk"
  namespace: "jenkins"
spec:
  containers:
  - args:
    - "cat"
    command:
    - "/usr/bin/env"
    - "sh"
    - "-c"
    env:
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "docker-registry.local/goorange/ci/golang:3.0.4"
    imagePullPolicy: "IfNotPresent"
    name: "golang"
    resources:
      limits:
        memory: "5000Mi"
      requests:
        memory: "5000Mi"
        cpu: "800m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  - args:
    - "********"
    - "test-test-go-25-07z28-kbtj7-hjvxk"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-test-agent.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/opt/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "jenkins/inbound-agent:3131.vf2b_b_798b_ce99-3"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      requests:
        cpu: "220m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  dnsPolicy: "ClusterFirstWithHostNet"
  hostNetwork: true
  imagePullSecrets:
  - name: "regsecret"
  nodeSelector:
    k8s.cluster.com/build-tenant: "jenkins-agents-nodegroup"
  restartPolicy: "Never"
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
  serviceAccountName: "jenkins-service-account"
  tolerations:
  - effect: "NoSchedule"
    key: "jenkins"
    operator: "Equal"
    value: "true"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - configMap:
      name: "jenkins-agent-configs"
      optional: false
    name: "volume-2"
  - emptyDir:
      medium: "Memory"
    name: "volume-1"
  - csi:
      driver: "secrets-store.csi.k8s.io"
      readOnly: true
      volumeAttributes:
        secretProviderClass: "jenkins-secret"
    name: "jenkins-secrets"
  - hostPath:
      path: "/opt/agent"
    name: "workspace-volume"

Pipeline looks like this:

pipeline {
    agent {
        kubernetes {
            inheritFrom 'golang'
            defaultContainer 'golang'
            showRawYaml true
        }
    }

    stages {
        stage('PATH') {
            steps {
                container('golang') {
                    sh "go version"
                    sh "which go"
                    sh "echo $PATH"
                    sh "echo ${PATH}"
                    sh 'echo ${PATH}'
                }
            }
        }
    }
}

Output:

+ go version
go version go1.20.6 linux/amd64
[Pipeline] sh
+ which go
/usr/local/go/bin/go
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin

What is wrong? Why is sh "echo $PATH" printing the PATH from jnlp-container? Or is it from Jenkins controller?!

saper · March 16, 2026, 7:13pm

I think this is a result of a String interpolation differences and this would mean that, if you say

echo "${PATH}"

this gets interpolated by Groovy (which is Java which is running probably within JNLP agent) so it takes the PATH variable from that environment. If you say

echo '${PATH}'

This will be left by Groovy alone and echo '${PATH}' will be sent to your shell running in the golang container.

As your printenv example shows, the underlying shell environment does contains the go path.

We are using ENV PATH=/local/user/.local/bin:${PATH} in our containers and it works as it should last time I checked…

Topic		Replies	Views
How to Override PATH environment variable? Ask a question question	3	18610	March 14, 2022
Docker, node not found in jenkins agent Ask a question question	14	5461	May 5, 2024
Jenkins Serverless Architecture Using Jenkins	0	874	April 14, 2023
Jenkins not picking custom JNLP Image in Kubernetes cluster Using Jenkins	1	568	January 7, 2026
Jenkins agent.jar when running agents in Kubernetes? Ask a question	4	1355	February 21, 2022

PATH variable is overwritten by jnlp container

Related topics