Unable to run containerized agents using Docker Cloud provider plugin

I can’t seem to get containerized agents working in Jenkins. Can anybody help?

Jenkins setup:

Jenkins and plugins versions report

Environment
Jenkins: 2.561
OS: Linux - 6.19.13-200.fc43.x86_64
Java: 21.0.10 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
ansicolor:536.v13fa_b_860c267
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6-193.vf028a_770a_fa_c
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
bootstrap5-api:5.3.8-1024.v127320880c60
bouncycastle-api:2.30.1.84-291.v9f17b_21896e2
branch-api:2.1280.v0d4e5b_b_460ef
build-name-setter:2.5.1
build-timeout:1.40
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
cloud-stats:377.vd8a_6c953e98e
cloudbees-folder:6.1100.ve9eed61d16c4
commons-compress-api:1.28.0-3
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-218.va_61573470393
config-file-provider:1006.vc7366c201f57
credentials:1502.v5c95e620ddfe
credentials-binding:719.v80e905ef14eb_
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:472.vee120e23d3a_c
docker-java-api:3.7.1-136.v5d70f77a_c3d6
docker-plugin:1316.v75635a_002b_0a_
durable-task:664.v2b_e7a_dfff66c
echarts-api:6.0.0-1281.vd3d21a_1ca_cb_4
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
font-awesome-api:7.2.0-983.v3f63c34eddb_9
git:5.10.1
git-client:6.6.0
github:1.46.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1967.vdea_d580c1a_b_a_
github-oauth:685.v53b_070455063
global-slack-notifier:1.5
gravatar:214.222.ve3d58ca_9015c
gson-api:2.13.2-198.v45e4a_55d9b_a_a_
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson-annotations2-api:2.21-7.v4777a_f3a_a_d47
jackson2-api:2.21.2-436.v29efdb_7418ff
jackson3-api:3.1.2-73.v3e5485d8b_148
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jjwt-api:0.13.0-141.vd58b_a_9592b_6c
joda-time-api:2.14.1-187.vdf2def02b_8a_1
jquery3-api:3.7.1-682.vfa_cdce169929
json-api:20251224-185.v0cc18490c62c
json-path-api:3.0.0-218.vcd4dd1355de2
junit:1403.vd9d1413fd205
mailer:534.v1b_36f5864073
matrix-auth:3.2.9
matrix-project:870.v9db_fcfc2f45b_
metrics:4.2.37-494.v06f9a_939d33a_
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
next-executions:517.vc2c2ca_1b_c808
okhttp-api:5.3.2-200.vedb_720a_cf1f8
oss-symbols-api:442.v99039087229b_
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-graph-analysis:254.v0f63a_a_447dca_
pipeline-graph-view:844.va_1e08a_05a_90f
pipeline-groovy-lib:797.v90ea_a_9b_e45a_0
pipeline-input-step:551.vdff487c5998c
pipeline-maven:1672.v1b_d4c0435b_20
pipeline-maven-api:1672.v1b_d4c0435b_20
pipeline-milestone-step:152.v6e22b_8cfc66c
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.41
pipeline-stage-step:345.va_96187909426
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.41
plain-credentials:199.v9f8e1f741799
plugin-util-api:7.1330.v47b_46ee2047a_
prism-api:1.30.0-720.v1eb_7496954b_3
scm-api:728.vc30dcf7a_0df5
script-security:1399.ve6a_66547f6e1
slack:795.v4b_9705b_e6d47
snakeyaml-api:2.5-149.v72471e9c6371
snakeyaml-engine-api:3.0.1-5.vd98ea_ff3b_92e
ssh-credentials:372.va_250881b_08cd
ssh-slaves:3.1097.v868116049892
sshd:3.384.vc89b_5e138cf9
structs:362.va_b_695ef4fdf9
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
woodstox-core-api:7.1.1-1.v4d297985f397
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1413.v2ff1a_5e720fa_
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4285.v8df38f05c3c5
workflow-durable-task-step:1475.ved562f6ec8b_3
workflow-job:1573.v1465f6f78810
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:724.v538c2362b_dfb_
workflow-support:1015.v785e5a_b_b_8b_22

What Operating System are you using (both controller, and any agents involved in the problem)?

I’m using Fedora 43 for the controller, and attempting to use docker.io/jenkins/agent:latest-jdk21 for the agents.

Reproduction steps

  1. Install docker cloud plugin
  2. Configure cloud plugin using unix socket for docker (or podman; I’m using podman)
  3. Configure agent templates using default settings, and the docker.io/jenkins/agent:latest-jdk21 image
  4. Configure built-in node to have 0 executors
  5. Start a job

(I can provide more detailed steps for these, as needed, but I’m not doing anything abnormal, so nothing stands out as relevant.)

Expected Results

I would be able to configure a Docker cloud using the docker cloud plugin, and dynamically launch agents to run jobs.

Actual Results

The controller fails to launch the agents. I get this output when I check the agent status at https://myjenkinshost/computer/jenkins-agent-jdk21-000002hnj6bjw/log and the jobs never run:

Connecting to docker container 52604a47458130fb1e955299fb2421c81720929630b21aa53dac9c24435e1fcf, running command java -jar /home/jenkins/remoting-3355.v388858a_47b_33.jar -noReconnect -noKeepAlive -agentLog /home/jenkins/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.multiplexed-stream
Connection: Upgrade
Upgrade: tcp
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins
Also:   java.lang.Throwable: launched here
		at hudson.slaves.SlaveComputer._connect(SlaveComputer.java:287)
		at hudson.model.Computer.connect(Computer.java:442)
		at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy.start(DockerOnceRetentionStrategy.java:145)
		at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.strategy.DockerOnceRetentionStrategy.start(DockerOnceRetentionStrategy.java:49)
		at hudson.model.AbstractCIBase.createNewComputerForNode(AbstractCIBase.java:189)
		at hudson.model.AbstractCIBase.updateNewComputer(AbstractCIBase.java:218)
		at jenkins.model.Jenkins.updateNewComputer(Jenkins.java:1595)
		at jenkins.model.Nodes.handleAddedNode(Nodes.java:191)
		at jenkins.model.Nodes.addNode(Nodes.java:185)
		at jenkins.model.Jenkins.addNode(Jenkins.java:2192)
		at PluginClassLoader for docker-plugin//io.jenkins.docker.DockerTransientNode.robustlyAddToJenkins(DockerTransientNode.java:448)
		at PluginClassLoader for docker-plugin//com.nirima.jenkins.plugins.docker.DockerCloud$1.run(DockerCloud.java:421)
		at jenkins.util.ContextResettingExecutorService.lambda$wrap$0(ContextResettingExecutorService.java:26)
		at jenkins.security.ImpersonatingExecutorService.lambda$wrap$0(ImpersonatingExecutorService.java:66)
		at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
org.newsclub.net.unix.ConnectionResetSocketException: Connection reset by peer
	at PluginClassLoader for docker-plugin//org.newsclub.net.unix.NativeUnixSocket.read(Native Method)
	at PluginClassLoader for docker-plugin//org.newsclub.net.unix.AFSocketImpl$AFInputStreamImpl.read(AFSocketImpl.java:669)
	at PluginClassLoader for docker-plugin//io.jenkins.docker.client.DockerMultiplexedInputStream.readInternal(DockerMultiplexedInputStream.java:62)
	at PluginClassLoader for docker-plugin//io.jenkins.docker.client.DockerMultiplexedInputStream.read(DockerMultiplexedInputStream.java:32)
	at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:491)
	at hudson.remoting.ChannelBuilder.build(ChannelBuilder.java:437)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:441)
	at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:408)
	at PluginClassLoader for docker-plugin//io.jenkins.docker.connector.DockerComputerAttachConnector$DockerAttachLauncher.launch(DockerComputerAttachConnector.java:354)
	at hudson.slaves.DelegatingComputerLauncher.launch(DelegatingComputerLauncher.java:64)
	at PluginClassLoader for docker-plugin//io.jenkins.docker.connector.DockerDelegatingComputerLauncher.launch(DockerDelegatingComputerLauncher.java:46)
	at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)
	at jenkins.util.ContextResettingExecutorService.lambda$wrap$1(ContextResettingExecutorService.java:41)
	at jenkins.security.ImpersonatingExecutorService.lambda$wrap$1(ImpersonatingExecutorService.java:75)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Anything else?

For some reason, I was able to get this to work the very first time I tried it, but after shutting down to take a backup of the working system, it never worked again.

Also, I am using podman instead of docker, but as far as I can tell, that has no impact on the configuration and works just as well as docker would. When I run commands manually using the same podman socket that I have configured for Jenkins.

When I attach to the still-running containers that Jenkins started and said failed, I can run the remoting agent jar that Jenkins copied to the container, and I don’t see any errors. I can’t tell if the stack trace above is coming from the docker plugin or from the agent jar, but I think it’s coming from the docker plugin.

After much troubleshooting on my own, I tried to report this at Unable to run containerized agents using Docker Cloud provider plugin · Issue #26696 · jenkinsci/jenkins · GitHub , but the ticket was immediately closed, suggesting I try here. At this point, I’m pretty sure there’s a bug in either Jenkins or the Docker Cloud plugin, because the issue seems to be something to do with launching the agents. The error message suggested that it was probably a bug in Jenkins, but since I’m not getting any help as a bug report, I’m hoping I can make some progress here.

You say that you’re not doing anything abnormal, but before that, you say:

In that sentence, I see two things that are different than the way that I’ve configured my Docker on Fedora 43 to provide cloud agents.

I use a network connection with an X.509 certificate and I use Docker version 29.4.0, build 1.fc43.

My controller sometimes runs on Fedora 43 in a Docker container and it sometimes runs on Ubuntu in a Docker container.

I’ve configured the Docker service with systemctl edit docker to look like this:

[Unit]
Description=Docker Application Container Engine on fedora-a

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd \
    -H fd:// \
    -H 0.0.0.0:2376 \
    --containerd=/run/containerd/containerd.sock \
    --selinux-enabled \
    --userland-proxy-path /usr/bin/docker-proxy \
    --tlsverify \
    --tlscacert=/root/docker-cert/ca.pem \
    --tlscert=/root/docker-cert/server-cert.pem \
    --tlskey=/root/docker-cert/server-key.pem \
    --init-path /usr/bin/tini-static

In the /root/docker directory, I have a README that says:

The certificates and keys in this folder are used by dockerd to grant access to Docker on this computer from other computers. My Jenkins controllers use the certificates to connect so that they can run cloud agents.

When adding a new cloud with the Docker plugin, I use the following files from this directory:

  • key.pem
  • cert.pem
  • ca.pem

Generated based on Protect the Docker daemon socket | Docker Docs

The plugin documentation does not mention podman. I don’t test the plugin with podman. I’m not aware of anyone using the plugin with podman.

The plugin documentation mentions the Unix domain socket, but I use a network connection because my controller is not always on the same host as the Docker server.

When I say I’m not doing anything unusual, I mean with the Jenkins / plugin configuration. I use the default values for everything and am using the suggested agent images for the version of Java I’m using.

The plugin configuration has the following tooltip for the “Docker Host Uri”: typically unix:///var/run/docker.sock. So, I’m basically using that, but the podman socket instead of the docker socket. Podman is compatible in this way, and I have no problems communicating with podman or attaching to it, whatsoever. The issue seems to be in the communication with the agent after the container is launched, after it has copied the remoting agent jar to the container, and after it has attached and run the agent jar.

I am limited to running a single node, so the socket communication is just fine for me. I don’t need to communicate over a network using TCP.

Importantly, this all worked the first time I set it up, but after a reboot, Jenkins started complaining that it couldn’t launch the agents, but the container service is still working just fine, without any complaints. I can see the containers that Jenkins has launched, and as far as podman is concerned, they are healthy and running just fine. The issue seems to be with Jenkins communicating with the agent.

If it helps, here are my systemd unit files for podman, but they are pretty basic. I just copied the ones that ship with the RPM and added lines to run as the jenkins user:

/etc/systemd/system/jenkins-podman.service:

[Unit]
Description=Podman API Service for Jenkins daemon
Requires=jenkins-podman.socket
After=jenkins-podman.socket
Documentation=man:podman-system-service(1)
StartLimitIntervalSec=0

[Service]
User=jenkins
Group=jenkins
Delegate=true
Type=exec
KillMode=process
Environment=LOGGING="--log-level=info"
ExecStart=/usr/bin/podman $LOGGING system service

[Install]
WantedBy=default.target

/etc/systemd/system/jenkins-podman.socket:

[Unit]
Description=Podman API Socket for Jenkins daemon
Documentation=man:podman-system-service(1)

[Socket]
SocketUser=jenkins
SocketGroup=jenkins
ListenStream=%t/jenkins-podman/podman.sock
SocketMode=0660

[Install]
WantedBy=sockets.target

I could have set up podman to listen indefinitely on the socket instead of setting up a socket listener this way and launching it on demand, but I wanted to keep as close to the built-in RPM-provided unit files as possible, to minimize errors on my part.