PATH variable is overwritten by jnlp container

mohanisch · October 7, 2022, 8:43pm

Hello,

We are using Jenkins with kubernetes plugin.
In Jenkins, various agents are set up, all with a custom image. In one image, for example, golang is installed and the PATH is adjusted accordingly:

ENV GOPATH /home/jenkins/go
ENV PATH $PATH:/usr/local/go/bin:$GOPATH/bin

Now we have noticed that in the pipeline $PATH from the golang-image is overwritten with the $PATH from the jnlp container.

Pipeline:

pipeline {
    agent {
        kubernetes {
		    inheritFrom 'golang'
		    defaultContainer 'go'
        }
    }

    stages {
        stage ('test'){
            steps {
                sh "echo ${PATH}"
                sh "echo ${env.PATH}"
                println Path
                sh "printenv"
            }
        }
    }
}

Result from job log:

+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] echo
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ printenv
....
PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
....

Similar issue we have with a java agent. We installed two java-versions and set im Dockerfile:

ENV JAVA_HOME_18 /usr/lib/jvm/jdk-18
ENV JAVA_HOME_19 /usr/lib/jvm/jdk-19

In pipeline:

sh "echo $env.JAVA_HOME_19"

has no result.

What is wrong?

Thanks in advance!
Marco

mohanisch · October 9, 2022, 1:35pm

Okay, I got it now.
We are migrating from ssh-agents to kubernetes agents.
Before, we were able to use each environment variable from agent (backed into image).

How can we still use the environment of the agents?

saper · October 9, 2022, 2:20pm

What you could try is to run two containers in one pod, jnlp-agent and your golang stuff separately. jnlp-agent is pretty cut down and meant only to run the agent itself.

mohanisch · October 24, 2022, 9:32am

@saper sorry for late reply.

The jnlp container is created automatically by kubernetes plugin.

We are facing more and more issues… most of them are related to PATH.

While running the pipeline we cannot add something to PATH like

PATH=PATH:/path/to/some/bin

This is for example added es ENV in the Dockerfile but this path is not populated.

Echo on PATH has a different output as printev for example.

What is wrong?

mohanisch · October 24, 2022, 10:02am

Here one example. $PATH is configured as environment variable in pod.

Pipeline:

pipeline {
    agent {
        kubernetes {
  		    inheritFrom 'agent-build'
  		    defaultContainer 'build'
        }
    }
    stages {
        stage('TEST') {
            steps {
                echo "Hello world"
                echo "-------------"
                sh 'echo $PATH'
                echo "-------------"
                sh "PATH=$PATH:/some/path/to/bin"
                sh "echo $PATH"
                echo "-------------"
                sh """
                    PATH=$PATH:/some/path/to/bin
                    echo $PATH
                """
            }
        }
    }
}

Output:

[Pipeline] stage
[Pipeline] { (TEST)
[Pipeline] echo
Hello world
[Pipeline] echo
-------------
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

mohanisch · August 2, 2023, 7:40am

This issue is still existing… Here is the print how the pod got created:

apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins-test.jenkins.svc.cluster.local:8080/job/TEST/job/test-go/25/"
    runUrl: "job/TEST/job/test-go/25/"
  labels:
    jenkins/jenkins-test-jenkins-agent: "true"
    jenkins/label-digest: "bf10494b3f401f3387218f696f5b0565e9ca22cd"
    jenkins/label: "TEST_test-go_25-07z28"
  name: "test-test-go-25-07z28-kbtj7-hjvxk"
  namespace: "jenkins"
spec:
  containers:
  - args:
    - "cat"
    command:
    - "/usr/bin/env"
    - "sh"
    - "-c"
    env:
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "docker-registry.local/goorange/ci/golang:3.0.4"
    imagePullPolicy: "IfNotPresent"
    name: "golang"
    resources:
      limits:
        memory: "5000Mi"
      requests:
        memory: "5000Mi"
        cpu: "800m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  - args:
    - "********"
    - "test-test-go-25-07z28-kbtj7-hjvxk"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-test-agent.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/opt/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "jenkins/inbound-agent:3131.vf2b_b_798b_ce99-3"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      requests:
        cpu: "220m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  dnsPolicy: "ClusterFirstWithHostNet"
  hostNetwork: true
  imagePullSecrets:
  - name: "regsecret"
  nodeSelector:
    k8s.cluster.com/build-tenant: "jenkins-agents-nodegroup"
  restartPolicy: "Never"
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
  serviceAccountName: "jenkins-service-account"
  tolerations:
  - effect: "NoSchedule"
    key: "jenkins"
    operator: "Equal"
    value: "true"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - configMap:
      name: "jenkins-agent-configs"
      optional: false
    name: "volume-2"
  - emptyDir:
      medium: "Memory"
    name: "volume-1"
  - csi:
      driver: "secrets-store.csi.k8s.io"
      readOnly: true
      volumeAttributes:
        secretProviderClass: "jenkins-secret"
    name: "jenkins-secrets"
  - hostPath:
      path: "/opt/agent"
    name: "workspace-volume"

Pipeline looks like this:

pipeline {
    agent {
        kubernetes {
            inheritFrom 'golang'
            defaultContainer 'golang'
            showRawYaml true
        }
    }

    stages {
        stage('PATH') {
            steps {
                container('golang') {
                    sh "go version"
                    sh "which go"
                    sh "echo $PATH"
                    sh "echo ${PATH}"
                    sh 'echo ${PATH}'
                }
            }
        }
    }
}

Output:

+ go version
go version go1.20.6 linux/amd64
[Pipeline] sh
+ which go
/usr/local/go/bin/go
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin

What is wrong? Why is sh "echo $PATH" printing the PATH from jnlp-container? Or is it from Jenkins controller?!

Topic		Replies	Views
JNLP default image override at top level. Is it possible? Using Jenkins question	1	561	March 1, 2022
Jenkins not picking custom JNLP Image in Kubernetes cluster Using Jenkins	0	354	April 17, 2024
How to Override PATH environment variable? Ask a question question	3	16417	March 14, 2022
Pod Template and Custom Agent Images Ask a question question , docker , kubernetes	5	279	July 10, 2024
Cannot update PATH env on Jenkins node Ask a question	2	42	October 16, 2024

PATH variable is overwritten by jnlp container

Related topics