PATH variable is overwritten by jnlp container

mohanisch · October 7, 2022, 8:43pm

Hello,

We are using Jenkins with kubernetes plugin.
In Jenkins, various agents are set up, all with a custom image. In one image, for example, golang is installed and the PATH is adjusted accordingly:

ENV GOPATH /home/jenkins/go
ENV PATH $PATH:/usr/local/go/bin:$GOPATH/bin

Now we have noticed that in the pipeline $PATH from the golang-image is overwritten with the $PATH from the jnlp container.

Pipeline:

pipeline {
    agent {
        kubernetes {
		    inheritFrom 'golang'
		    defaultContainer 'go'
        }
    }

    stages {
        stage ('test'){
            steps {
                sh "echo ${PATH}"
                sh "echo ${env.PATH}"
                println Path
                sh "printenv"
            }
        }
    }
}

Result from job log:

+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] echo
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ printenv
....
PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
....

Similar issue we have with a java agent. We installed two java-versions and set im Dockerfile:

ENV JAVA_HOME_18 /usr/lib/jvm/jdk-18
ENV JAVA_HOME_19 /usr/lib/jvm/jdk-19

In pipeline:

sh "echo $env.JAVA_HOME_19"

has no result.

What is wrong?

Thanks in advance!
Marco

mohanisch · October 9, 2022, 1:35pm

Okay, I got it now.
We are migrating from ssh-agents to kubernetes agents.
Before, we were able to use each environment variable from agent (backed into image).

How can we still use the environment of the agents?

saper · October 9, 2022, 2:20pm

What you could try is to run two containers in one pod, jnlp-agent and your golang stuff separately. jnlp-agent is pretty cut down and meant only to run the agent itself.

mohanisch · October 24, 2022, 9:32am

@saper sorry for late reply.

The jnlp container is created automatically by kubernetes plugin.

We are facing more and more issues… most of them are related to PATH.

While running the pipeline we cannot add something to PATH like

PATH=PATH:/path/to/some/bin

This is for example added es ENV in the Dockerfile but this path is not populated.

Echo on PATH has a different output as printev for example.

What is wrong?

mohanisch · October 24, 2022, 10:02am

Here one example. $PATH is configured as environment variable in pod.

Pipeline:

pipeline {
    agent {
        kubernetes {
  		    inheritFrom 'agent-build'
  		    defaultContainer 'build'
        }
    }
    stages {
        stage('TEST') {
            steps {
                echo "Hello world"
                echo "-------------"
                sh 'echo $PATH'
                echo "-------------"
                sh "PATH=$PATH:/some/path/to/bin"
                sh "echo $PATH"
                echo "-------------"
                sh """
                    PATH=$PATH:/some/path/to/bin
                    echo $PATH
                """
            }
        }
    }
}

Output:

[Pipeline] stage
[Pipeline] { (TEST)
[Pipeline] echo
Hello world
[Pipeline] echo
-------------
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
[Pipeline] sh
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] echo
-------------
[Pipeline] sh
+ PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/some/path/to/bin
+ echo /usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

mohanisch · August 2, 2023, 7:40am

This issue is still existing… Here is the print how the pod got created:

apiVersion: "v1"
kind: "Pod"
metadata:
  annotations:
    buildUrl: "http://jenkins-test.jenkins.svc.cluster.local:8080/job/TEST/job/test-go/25/"
    runUrl: "job/TEST/job/test-go/25/"
  labels:
    jenkins/jenkins-test-jenkins-agent: "true"
    jenkins/label-digest: "bf10494b3f401f3387218f696f5b0565e9ca22cd"
    jenkins/label: "TEST_test-go_25-07z28"
  name: "test-test-go-25-07z28-kbtj7-hjvxk"
  namespace: "jenkins"
spec:
  containers:
  - args:
    - "cat"
    command:
    - "/usr/bin/env"
    - "sh"
    - "-c"
    env:
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "docker-registry.local/goorange/ci/golang:3.0.4"
    imagePullPolicy: "IfNotPresent"
    name: "golang"
    resources:
      limits:
        memory: "5000Mi"
      requests:
        memory: "5000Mi"
        cpu: "800m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  - args:
    - "********"
    - "test-test-go-25-07z28-kbtj7-hjvxk"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "jenkins-test-agent.jenkins.svc.cluster.local:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "GRADLE_USER_HOME"
      value: "/home/jenkins/.gradle"
    - name: "GRADLE_CACHE_USR"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-user"
          name: "gradle-cache-secret"
          optional: false
    - name: "GRADLE_CACHE_PSW"
      valueFrom:
        secretKeyRef:
          key: "gradle-cache-password"
          name: "gradle-cache-secret"
          optional: false
    - name: "JENKINS_NAME"
      value: "test-test-go-25-07z28-kbtj7-hjvxk"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/opt/agent"
    - name: "JENKINS_URL"
      value: "http://jenkins-test.jenkins.svc.cluster.local:8080/"
    - name: "GRADLE_HOME"
      value: "/usr/local/gradle"
    image: "jenkins/inbound-agent:3131.vf2b_b_798b_ce99-3"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      requests:
        cpu: "220m"
    tty: true
    volumeMounts:
    - mountPath: "/home/jenkins/.gradle/caches"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/.gradle/gradle.properties"
      name: "volume-2"
      readOnly: false
      subPath: "gradle.properties"
    - mountPath: "/opt/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/opt/agent"
  dnsPolicy: "ClusterFirstWithHostNet"
  hostNetwork: true
  imagePullSecrets:
  - name: "regsecret"
  nodeSelector:
    k8s.cluster.com/build-tenant: "jenkins-agents-nodegroup"
  restartPolicy: "Never"
  securityContext:
    fsGroup: 1000
    runAsUser: 1000
  serviceAccountName: "jenkins-service-account"
  tolerations:
  - effect: "NoSchedule"
    key: "jenkins"
    operator: "Equal"
    value: "true"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - configMap:
      name: "jenkins-agent-configs"
      optional: false
    name: "volume-2"
  - emptyDir:
      medium: "Memory"
    name: "volume-1"
  - csi:
      driver: "secrets-store.csi.k8s.io"
      readOnly: true
      volumeAttributes:
        secretProviderClass: "jenkins-secret"
    name: "jenkins-secrets"
  - hostPath:
      path: "/opt/agent"
    name: "workspace-volume"

Pipeline looks like this:

pipeline {
    agent {
        kubernetes {
            inheritFrom 'golang'
            defaultContainer 'golang'
            showRawYaml true
        }
    }

    stages {
        stage('PATH') {
            steps {
                container('golang') {
                    sh "go version"
                    sh "which go"
                    sh "echo $PATH"
                    sh "echo ${PATH}"
                    sh 'echo ${PATH}'
                }
            }
        }
    }
}

Output:

+ go version
go version go1.20.6 linux/amd64
[Pipeline] sh
+ which go
/usr/local/go/bin/go
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[Pipeline] sh
+ echo /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/jenkins/go/bin:/opt/java/openjdk/bin

What is wrong? Why is sh "echo $PATH" printing the PATH from jnlp-container? Or is it from Jenkins controller?!

Topic		Replies	Views
JNLP default image override at top level. Is it possible? Using Jenkins question	1	459	March 1, 2022
Jenkins not picking custom JNLP Image in Kubernetes cluster Using Jenkins	0	63	April 17, 2024
Cannot use Workspace PVC (java.io.IOException: Remote call on JNLP4-connect connection from) Using Jenkins	0	1134	February 14, 2022
Docker plugin: How to create build image still providing JDK8 with Jenkins 2.361.1? Ask a question	4	1734	September 24, 2022
Using different Java versions for agents and jobs Ask a question question	2	419	February 14, 2024

PATH variable is overwritten by jnlp container

Related Topics