Docker, node not found in jenkins agent

Hi, there

how to fix it? Thanks a lot.

Env Version

here is my agent host env:

jenkins@saller:~$ go version
go version go1.20.5 linux/amd64
jenkins@saller:~$ node -v
v18.16.0
jenkins@saller:~$ docker -v
Docker version 20.10.21, build 20.10.21-0ubuntu1~22.04.3
jenkins@saller:~$

Error log

FYI.

Controller node connects with agent by jenkins/ssh-agent image

docker run -d --rm --name=agent1 -p 2222:22 -e "JENKINS_AGENT_SSH_PUBKEY=YOUR_PUB_KEY" jenkins/ssh-agent:alpine

I found this:

It seems that Jenkins ssh to the ssh-agent container, in which node, docker, and go do not exist.

How can I fix it?

  • To install them to container?
  • Or only run agent.jar on agent host?

Could someone give me some help? Thanks in advance.

I don’t know if this is the root cause of your problem, but when using ssh you don’t have the same environment, the sourced configuration files aren’t the same.
When logging in via SSH, the environment variables and settings are typically sourced from different files such as .bash_profile , .bashrc , or .profile , depending on your shell configuration. These files are sourced during an interactive login session and may set up additional environment variables or modify the default PATH.

Hi, @poddingue

Question 1

If using jenkins/ssh-agent image, the command(e.g. node, docker, go, etc.) used by job is from the host not the container, right?

Question 2

I tried to ssh host directly instead of jenkins/ssh-agent image used by jenkins agent config, agent can be online. But, the command node, docker, go still not found.
Here is my host config:

jenkins@saller:~$ tail -n10 ~/.bashrc
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

. /etc/profile

jenkins@saller:~$ cat /etc/profile.d/local.sh
export PATH=$PATH:/usr/local/go/bin
export PATH=$PATH:/usr/local/node/bin

export PATH=$PATH:$HOME/.local/bin
export PATH=$PATH:$HOME/go/bin

jenkins@saller:~$

As far as I understand, the commands you use are the ones from the jenkins/ssh-agent image, and not the commands from the host itself.
If you need node , docker or any other command that is not part of the jenkins/ssh-agent image, you would have to create your own image based on that.

I think I completely misunderstood your first problem, sorry for that.

So, If I directly ssh agent host instead of ssh-agent image, the jenkins job will use the commands from the host, is it right?

If it’s right, I still encounter the issue: go, node not found

So, If I directly ssh agent host instead of ssh-agent image, the jenkins job will use the commands from the host, is it right?

Yes, if they are available through ssh (in the $PATH) to Jenkins.
While being logged on the host, issue which node for example, so you get the full path of the binary.
Then, in your Jenkinsfile, use that full path instead of just the command name, and see if the command runs.

Ok, Thanks a lot, @poddingue
I will try it later, and then give you feedback.
Thanks.

1 Like

On ci.jenkins.io we assign labels to the agents so that the label represents a capability of the agent. An agent with the label docker is able to perform docker operations inside the agent. An agent with the label maven-17-windows is a Windows computer that can run Apache Maven with Java 17.

Those labels are referenced from jobs so that the job declares the type of capabilities it needs, without declaring the type of agent (ephemeral, static, cloud, non-cloud, etc.).

I use a similar setup on my controller and it allows me to use different implementations to run the same jobs as run on ci.jenkins.io. I have static agents while ci.jenkins.io uses ephemeral agents, yet the jobs run as well on my controller as they do on ci.jenkins.io.

I use the platform labeler plugin to automatically assign labels to agents and the implied labels plugin to define labels for agents based on labels that are already assigned to those agents. The combination of those two plugins allows me to automatically assign labels for most capabilities.

Sorry, I don’t understand what you say. ><

Hi, @poddingue

I found: If the commands being in /usr/bin/(such as docker, java who are installed by me) can be found by jenkins agent.
So, one way my workround is:

  • create symbolic link for node, go, etc.
    • sudo ln -s $(which go) /usr/bin/go
    • sudo ln -s $(which node) /usr/bin/node

Question

I don’t know why it doesn’t load /etc/profile, if it load it, I think it can work fine.

FYI

Here is my /etc/profile.d/local.sh config:

export PATH=$PATH:/usr/local/go/bin
export PATH=$PATH:/usr/local/node/bin

export PATH=$PATH:$HOME/.local/bin
export PATH=$PATH:$HOME/go/bin

1 Like

It still need a way to load /etc/profile or just load $HOME/go/bin, otherwise the go binay cannot be load(There will be so many go binaries installed when the job running)

Final Solution

Add System Environment Variables on Jenkins Controller Node

2 Likes

Nice, I was thinking there was something fishy in your environment configuration.
Thanks a lot for your feedback. :+1:

My 50 cents about the topic.

I would like to create a golang ssh-agent so I created the following dockerfile to do that after reading the official ssh-agent alpine dockerfile.

FROM jenkins/ssh-agent:latest-alpine3.19

RUN wget https://go.dev/dl/go1.22.2.linux-amd64.tar.gz && \
    rm -rf /usr/local/go && \
    tar -C /usr/local -xzf go1.22.2.linux-amd64.tar.gz

ENV PATH="$PATH:/usr/local/go/bin"

# Alpine's ssh doesn't use $PATH defined in /etc/environment, so we define `$PATH` in `~/.ssh/environment`
RUN echo "PATH=${PATH}" >> ${JENKINS_AGENT_HOME}/.ssh/environment
1 Like