Unable to run jenkins build inside a docker agent using SSH connection method

[ShantanuParanjpe]

Hello Team,

Unable to run jenkins build inside a docker agent agent using SSH connection method.Getting below error:

[12/14/23 16:34:07] [SSH] Opening SSH connection to 192.168.56.110:32793.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: “Key exchange was not finished, connection is closed.”, retrying in 15 seconds. There are 10 more retries left.
Key exchange was not finished, connection is closed.
SSH Connection failed with IOException: “Key exchange was not finished, connection is closed.”, retrying in 15 seconds. There are 9 more retries left.
Key exchange was not finished, connection is closed.

Below is the Dockerfile used for creating the docker image:

FROM centos:7

# Install SSH server, Java, and other required packages
RUN yum -y install openssh-server java-11-openjdk-devel curl

# Create Jenkins user and set up directories
RUN useradd -m -s /bin/bash jenkins && *
** mkdir -p /home/jenkins/.ssh && *
** chown -R jenkins:jenkins /home/jenkins && **
** chmod 700 /home/jenkins/.ssh**

# Generate SSH keys for the Jenkins user
RUN ssh-keygen -t rsa -f /home/jenkins/.ssh/id_rsa

# Generate SSH host keys
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ‘’ && *
** ssh-keygen -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ‘’ && *
** ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ‘’**

# Copy the public key to the authorized_keys file for the Jenkins user
RUN cat /home/jenkins/.ssh/id_rsa.pub >> /home/jenkins/.ssh/authorized_keys && *
** chmod 600 /home/jenkins/.ssh/authorized_keys*

# SSH configuration for key-based authentication
RUN sed -i ‘s/#PubkeyAuthentication yes/PubkeyAuthentication yes/’ /etc/ssh/sshd_config && *
** sed -i ‘s/#PasswordAuthentication yes/PasswordAuthentication no/’ /etc/ssh/sshd_config*

# Expose SSH port
EXPOSE 22

CMD [“/usr/sbin/sshd”, “-D”]

Also, I am able to create the container using this dockerfile.

Can someone please help in checking this issue?

[NotMyFault (Alexander Brandes)]

Jenkins does no longer support CentOS 7 as per End of life operating systems

I’d recommend looking into one of the named alternatives listed in the blog post.

[ShantanuParanjpe]

Hello @NotMyFault,

Thanks for your response. Do I need to make any changes in the dockerfile content except changing the base image version?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

It seems like you are recreating portions of the jenkins/ssh-agent container image as your base container. Did those instructions not work for you?

[ShantanuParanjpe]

Hello Mark,
I tried to use the jenkins/ssh-agent in the jenkins agent agent setup, but getting below error:

SSHLauncher{host=‘192.168.56.110’, port=32912, credentialsId=‘docker-ssh’, jvmOptions=‘’, javaPath=‘’, prefixStartSlaveCmd=‘’, suffixStartSlaveCmd=‘’, launchTimeoutSeconds=60, maxNumRetries=10, retryWaitTime=15, sshHostKeyVerificationStrategy=hudson.plugins.sshslaves.verifiers.NonVerifyingKeyVerificationStrategy, tcpNoDelay=true, trackCredentials=true}
[12/17/23 18:46:58] [SSH] Opening SSH connection to 192.168.56.110:32912.
[12/17/23 18:46:58] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
[12/17/23 18:46:58] [SSH] Authentication failed.
Authentication failed.
[12/17/23 18:46:58] Launch failed - cleaning up connection
[12/17/23 18:46:58] [SSH] Connection closed.

I have attached the docker agent template details set using the jenkins console. In this case, I m running the docker host and the jenkins server on the same machine.

Could you please check?

[ShantanuParanjpe]

Hello Team,

I am using the bibinwilson/jenkins-agent docker to spin up docker agent agent to run a print a hello-world message using a freestyle jenkins job. When I run the build, it goes in a pending state forever even when I am using the same label in the docker cloud configuration and the jenkins freestyle job. Can someone please check?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

@ShantanuParanjpe

I am using the bibinwilson/jenkins-agent docker to spin up docker agent

There isn’t a bibinwilson/jenkins-agent container image on Dockerhub as far as I can see. The bibinwilson/jenkins-slave container image was last updated two years ago. Use one of the container agent images from the Jenkins project. They are actively maintained and regularly updated.

[ShantanuParanjpe]

Hii Mark,

Could you please suggest which image from jenkins project shall I use?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

@ShantanuParanjpe

Could you please suggest which image from jenkins project shall I use?

Same link as was posted earlier, Docker container jenknis/ssh-agent

[ShantanuParanjpe]

Hello Mark,
If I use this image I get the same authentication error as before. Not sure what changes do I need make in this case. Could please assist here?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

@ShantanuParanjpe

Not sure what changes do I need make in this case. Could please assist here?

You’ve not provided enough details so that others can duplicate your configuration. The details you’ve provided are using an unmaintained container image. Assisting without enough information is a frustrating experience for those who try to assist. Please provide clear steps of the exact changes you made so that others can duplicate your configuration exactly.

I took the following steps to confirm that I can use the jenkins/ssh-agent:5.22.0-alpine3.19-jdk21 container image with Jenkins 2.426.2 running with Java 21.

1. I added an ED 25519 SSH private key to my Jenkins 2.426.2 controller running with Java 21 and then on another machine, called “testing-b”, I started the ssh-agent container image listening on port 2200 and using the ssh public key like this:

docker run -d --rm --name=agent \
    --publish 2200:22 \
    -e "JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHoGWb80kSBywPtvWbgeng7itaw7Tu6d/nXaxIKTQ05Z" \
    jenkins/ssh-agent:5.22.0-alpine3.19-jdk21

2. I added the ssh private key associated with that ED25519 public key to my Jenkins 2.426.2 controller as a credential for the user “jenkins”
3. I configured a Jenkins agent on my Jenkins 2.426.2 controller with the agent home directory “/home/jenkins/agent”, the hostname “testing-b”, and the port 2200
4. I clicked “Launch agent” and confirmed that the agent connected correctly

I did not use the docker plugin to control that agent because I did not want the added complexity of configuring a network accessible docker daemon. Using a static agent is enough to show that the authentication works and the agent is connected.

[ShantanuParanjpe]

Hello @MarkEWaite ,
Apologies for not providing enough details. Below are the details:

I am using 2 machines:

Machine 1: Jenkins Controller
Machine 2: Docker Host

Requirement: I am trying to spin up docker agent agent on the docker host to run a print a hello-world message using a freestyle jenkins job. When I run the build, it goes in a pending state forever even when I am using the same label in the docker cloud configuration and the jenkins freestyle job. I am using the jenkins/ssh-agent docker image. Below is the screenshot jenkins cloud configuration:

I am using the SSH connection method to connect to the docker agent and I have stored the username and password as jenkins in the credential manager and used the same credentials in the docker agent cloud configuration to authenticate to the docker container.

Below is the freestyle job configuration:

When I click on Build now, the build goes in Pending state forever even when the label demo-docker is set in the docker cloud configuration.

I have already opened port range on the docker host(32768-60999)

I also tried to run a pipeline job for running the build remotely inside a docker agent but its getting stuck as below:

I also created my own docker image using below dockerfile but that too didn’t work:
[root@jenkins-server ~]# cat Dockerfile

Base image with necessary dependencies

FROM roboxes/centos8

RUN yum -y update
RUN yum -y remove java
RUN yum install -y
java-1.8.0-openjdk
java-1.8.0-openjdk-devel

Create Jenkins user and set up directories

RUN useradd -m -s /bin/bash jenkins &&
mkdir -p /home/jenkins/.ssh &&
chown -R jenkins:jenkins /home/jenkins &&
chmod 700 /home/jenkins/.ssh

#Set password for the jenkins user
RUN echo “jenkins:jenkins”|chpasswd

Generate SSH keys for the Jenkins user

#RUN ssh-keygen -t rsa -f /home/jenkins/.ssh/id_rsa

Copy the public key to the authorized_keys file for the Jenkins user

#RUN cat /home/jenkins/.ssh/id_rsa.pub >> /home/jenkins/.ssh/authorized_keys && \

chmod 600 /home/jenkins/.ssh/authorized_keys

Expose SSH port

EXPOSE 22

CMD [“/usr/sbin/sshd”, “-D”]

I am not able to see any docker agent being created here:

Can you please help in checking this?

Also, in my dockerfile I have made use of Password-based authentication instead of ssh keys. Is it mandatory to add a private key to the jenkins controller and use public key as below:
I added an ED 25519 SSH private key to my Jenkins 2.426.2 controller running with Java 21 and then on another machine, called “testing-b”, I started the ssh-agent container image listening on port 2200 and using the ssh public key like this
docker run -d --rm --name=agent
–publish 2200:22
-e “JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHoGWb80kSBywPtvWbgeng7itaw7Tu6d/nXaxIKTQ05Z”
jenkins/ssh-agent:5.22.0-alpine3.19-jdk21

Let me know if you need further details, will be glad to share the same

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

Have you confirmed that you can connect a static ssh agent that is running without using the docker plugin? See the detailed steps that I provided.

@ShantanuParanjpe

Also, in my dockerfile I have made use of Password-based authentication instead of ssh keys. Is it mandatory to add a private key to the jenkins controller and use public key

Yes, it is mandatory to use a private key when connecting a Jenkins ssh (outbound) agent. I’ve not heard of anyone using password based authentication to connect a Jenkins ssh agent.

[ShantanuParanjpe]

Hello Mark,

Thanks for your response…

As I have mentioned before, I have 2 machines:

Jenkins controller and Docker host. I want to spin up docker agent on the Docker host virtual machine.

Do I need to create the public key on the jenkins controller or the docker host machine.

Do I need to run the docker run command on the Jenkins controller machine?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

@ShantanuParanjpe

As I have mentioned before, I have 2 machines:

I have two machines in my configuration as well.

@ShantanuParanjpe

Jenkins controller and Docker host. I want to spin up docker agent on the Docker host virtual machine.

That is what I did in my example. I started a docker agent on the computer using the docker run command that I provided earlier.

@ShantanuParanjpe

Do I need to create the public key on the jenkins controller or the docker host machine?

The computer where the public key is created is almost always irrelevant. The private key must be stored in a Jenkins ssh private key credential. The matching public key is passed to the ssh-agent container as an argument. The Jenkins ssh private key credential is provided in the agent definition so that it can be used with the ssh-agent container.

@ShantanuParanjpe

Do I need to run the docker run command on the Jenkins controller machine?

No, the docker run command that I executed was on the agent machine, not on the controller. I don’t see any reason why you would need the docker command available on the controller, based on your description.

[ShantanuParanjpe]

Hello Mark,

As per your suggestion, I created a public-private key pair for the jenkins user on the Docker host machine and stored the private key on the jenkins credential manager. After doing this, I am getting the below error when I click on the launch agent. Below is the error I observed in the jenkins agent logs:

ERROR:

<b>ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:ssh-keys/method:publickey)
ERROR: Failed to authenticate as jenkins with credential=ssh-keys
java.io.IOException: Publickey authentication failed.</b>

[jenkins@docker-host .ssh]$ ls -ls
total 8
4 -rw-------. 1 jenkins jenkins 1823 Jan 15 09:57 id_rsa
4 -rw-r–r–. 1 jenkins jenkins 400 Jan 15 09:57 id_rsa.pub

Could you please assist here?

Thanks,

Shantanu

[MarkEWaite (Mark Waite)]

Please provide failure messages as text in the future rather than providing a screenshot. When you provide a failure message as a screenshot, you make it much more difficult for others to quote your message.

The screenshot includes a message that says something like:

PEM problem - it is of unknown type

That is likely an indication that the command you used to start the docker container that is running the jenkins/ssh-agent container has a mistake in the command line argument that provides the ssh public key to the container. In my earlier response I gave this example:

docker run -d --rm --name=agent \
    --publish 2200:22 \
    -e "JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHoGWb80kSBywPtvWbgeng7itaw7Tu6d/nXaxIKTQ05Z" \
    jenkins/ssh-agent:5.22.0-alpine3.19-jdk21

I suspect you may have used a mistaken command line like this:

docker run -d --rm --name=agent \
    --publish 2200:22 \
    -e "JENKINS_AGENT_SSH_PUBKEY=AAAAC3NzaC1lZDI1NTE5AAAAIHoGWb80kSBywPtvWbgeng7itaw7Tu6d/nXaxIKTQ05Z" \
    jenkins/ssh-agent:5.22.0-alpine3.19-jdk21

Notice the absence of the “ssh-ed25519” in the second case.

If that’s not the mistake you made, then you’ll need to diagnose the mistake by looking at the contents of /home/jenkins/.ssh/authorized_hosts in the ssh-agent container image to understand what data was placed in that file by your command line arguments to the command that started the ssh-agent container image.

[ShantanuParanjpe]

Hello Mark,

Thanks for your response…

I have passed below public key as a environment variable in the docker cloud template in jenkins

<img src="upload://wbQvGKEbl7ctRq4keQLJCumXPbD.png" alt="image.png" width="578" height="57">

Still I am getting the below error related to the public key authentication:

[01/15/24 14:25:59] [SSH] Opening SSH connection to [192.168.56.112:34026](http://192.168.56.112:34026).
[01/15/24 14:25:59] [SSH] WARNING: SSH Host Keys are not being verified. Man-in-the-middle attacks may be possible against this connection.
ERROR: Server rejected the 1 private key(s) for jenkins (credentialId:ssh-keys/method:publickey)
ERROR: Failed to authenticate as jenkins with credential=ssh-keys
java.io.IOException: Publickey authentication failed.
	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:349)
	at com.trilead.ssh2.Connection.authenticateWithPublicKey(Connection.java:472)
	at com.cloudbees.jenkins.plugins.sshcredentials.impl.TrileadSSHPublicKeyAuthenticator.doAuthenticate(TrileadSSHPublicKeyAuthenticator.java:110)
	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:431)
	at com.cloudbees.jenkins.plugins.sshcredentials.SSHAuthenticator.authenticate(SSHAuthenticator.java:468)
	at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:878)
	at hudson.plugins.sshslaves.SSHLauncher.lambda$launch$0(SSHLauncher.java:434)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Could not generate signature
	at com.trilead.ssh2.signature.KeyAlgorithm.generateSignature(KeyAlgorithm.java:43)
	at com.trilead.ssh2.auth.AuthenticationManager.authenticatePublicKey(AuthenticationManager.java:316)
	... 10 more
Caused by: java.security.SignatureException: Could not sign data
	at java.base/sun.security.rsa.RSASignature.engineSign(RSASignature.java:196)
	at java.base/java.security.Signature$Delegate.engineSign(Signature.java:1423)
	at java.base/java.security.Signature.sign(Signature.java:712)
	at com.trilead.ssh2.signature.KeyAlgorithm.generateSignature(KeyAlgorithm.java:41)
	... 11 more
Caused by: javax.crypto.BadPaddingException: RSA private key operation failed
	at java.base/sun.security.rsa.RSACore.crtCrypt(RSACore.java:209)
	at java.base/sun.security.rsa.RSACore.rsa(RSACore.java:130)
	at java.base/sun.security.rsa.RSASignature.engineSign(RSASignature.java:193)
	... 14 more

Also, do I need to create the authorized_keys file as it’s currently not present:
[jenkins@k8s-controller .ssh]$ logout
[root@k8s-controller .ssh]# cat /home/jenkins/.ssh/authorized_keys
cat: /home/jenkins/.ssh/authorized_keys: No such file or directory

Can you please check this further ?

Thanks,
Shantanu

[MarkEWaite (Mark Waite)]

@ShantanuParanjpe

Also, do I need to create the authorized_keys file as it’s currently not present:
[jenkins@k8s-controller .ssh]$ logout
[root@k8s-controller .ssh]# cat /home/jenkins/.ssh/authorized_keys
cat: /home/jenkins/.ssh/authorized_keys: No such file or directory

The agent container is the location where you need to check the contents of the authorized_keys file. When ssh connects from the controller to the agent, the ssh process on the agent uses the authorized_keys file to decide if the request from the controller should be allowed.

[ShantanuParanjpe]

Hello @MarkEWaite ,
Thanks for the help…

I am able to launch static agent using public/private key pair combination:

Started by user unknown or anonymous
Running as SYSTEM
Building remotely on demo-docker-0000v0yr45co9 on demo-docker (demo-docker) in workspace /home/jenkins/agent/workspace/test-build
[test-build] $ /bin/sh -xe /tmp/jenkins8920696603590887214.sh

• echo ‘This is run on docker agent’
  Finished: SUCCESS

I am using below Jenkinsfile to launch agent using pipeline job:
[root@k8s-controller node-js-project]# cat Jenkinsfile
pipeline {

agent {

// docker {
label ‘demo-docker’
// image ‘jenkins/ssh-agent’
// }
}

stages {
stage(‘Build’) {
steps {
script {
sh ‘hostname’
sh ‘ip a’
echo ‘Hello, this is the Build stage!’
}
}
}
}
}
I am getting below output:
Fetching upstream changes from origin

git --version # timeout=10
git --version # ‘git version 2.39.3’
git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
git fetch --tags --force --progress – origin +refs/heads/:refs/remotes/origin/ # timeout=10
Seen branch in repository origin/main
Seen 1 remote branch
Obtained node-js-project/Jenkinsfile from a8f01797c638188bd96e61d7b582e955c23a7827
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘demo-docker-00001fck43xhs on demo-docker’ is offline
Running on demo-docker-00001fck43xhs on demo-docker in /home/jenkins/agent/workspace/k8s-deploy-1_main
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
using credential git-creds
Cloning the remote Git repository
Using no checkout clone with sparse checkout.
Cloning with configured refspecs honoured and without tags
Cloning repository GitHub - ShantanuParanjpe/shaanrepo: Testing web pages
git init /home/jenkins/agent/workspace/k8s-deploy-1_main # timeout=10
Fetching upstream changes from GitHub - ShantanuParanjpe/shaanrepo: Testing web pages
git --version # timeout=10
git --version # ‘git version 2.43.0’
using GIT_ASKPASS to set credentials
git fetch --no-tags --force --progress – GitHub - ShantanuParanjpe/shaanrepo: Testing web pages +refs/heads/:refs/remotes/origin/ # timeout=10
git config remote.origin.url GitHub - ShantanuParanjpe/shaanrepo: Testing web pages # timeout=10
git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
Avoid second fetch
Checking out Revision a8f01797c638188bd96e61d7b582e955c23a7827 (main)
Commit message: “modified jenkinsfile”
git config core.sparsecheckout # timeout=10
git config core.sparsecheckout true # timeout=10
git read-tree -mu HEAD # timeout=10
git checkout -f a8f01797c638188bd96e61d7b582e955c23a7827 # timeout=10
git rev-list --no-walk 86b7b20f7c9947bfca438ec0b586a9b10e676794 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] script
[Pipeline] {
[Pipeline] sh

• hostname
  e28931fcd6e9
  [Pipeline] sh
• ip a
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  inet 127.0.0.1/8 scope host lo
  valid_lft forever preferred_lft forever
  12: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
  link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
  inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
  valid_lft forever preferred_lft forever
  [Pipeline] echo
  Hello, this is the Build stage!
  [Pipeline] }
  [Pipeline] // script
  [Pipeline] }
  [Pipeline] // stage
  [Pipeline] }
  [Pipeline] // withEnv
  [Pipeline] }
  [Pipeline] // node
  [Pipeline] End of Pipeline
  Finished: SUCCESS

The job is successful but I cant see any agent container getting created on docker host:
[root@k8s-controller node-js-project]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
069850759cd4 74e24ba53288 “docker-entrypoint.s…” 2 months ago Exited (255) 2 months ago 0.0.0.0:3000->3000/tcp, :::3000->3000/tcp sad_curran
27f4aa627453 Google Cloud console “/usr/local/bin/entr…” 2 months ago Exited (255) 6 weeks ago 127.0.0.1:32772->22/tcp, 127.0.0.1:32771->2376/tcp, 127.0.0.1:32770->5000/tcp, 127.0.0.1:32769->8443/tcp, 127.0.0.1:32768->32443/tcp minikube

[root@k8s-controller node-js-project]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Can you please check this?

Thanks,
Shantanu