Setting up windows cloud agents

Speaking of windows images, are there any good instructions on setting up a Windows Server based Docker node in the Manage Nodes and Clouds / Configure Clouds section? I have setup a windows server for running windows based jenkins nodes/agents and I am having a hard time finding all the proper configuration settings.

I split this off of Java 11 as default in Docker images - #2 as its not really about the change to java 11 inside agent images.

I am having a hard time finding all the proper configuration settings.

What do you mean by proper? Is something not working that you want fixed?

Maybe @dduportal can share what they did for jenkins infra and windows vms (though I think its mostly via azure)

There appears to be zero documentation on the setup of windows agents that I can find and when I attempt to run a build on one I receive the error bellow, which indicates that it is attempting to run /bin/sh, which of course does not exist on windows. There has to be someone somewhere has to be using the feature, yes? However, I can’t find any examples of actually configuring a windows docker node within Jenkins, so I am not sure if the error is because of a configuration issue, a bug, or a combination of both.


com.github.dockerjava.api.exception.BadRequestException: {“message”:“container a1966484e2336cbb68018153e85cfb938e5dd7d20f4a3b0725ab96e3a18f8153 encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {“CommandLine”:”/bin/sh",“User”:“jenkins”,“WorkingDirectory”:“C:\\Users\\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Current\\Bin\\Roslyn”},“EmulateConsole”:true,“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“ConsoleSize”:[0,0]}"}

Even outside of jenkins, running real windows docker images inside of windows is fairly new. For a long time windows would run a linux vm and run docker on that. That doesn’t excuse the lack of docs, but between that and people being volunteer it might explain it a bit.

I’m guessing we’ll end up creating a bug for it. I’m by no means an expert, but I can help narrow it down for those who are. For now more information is needed.

  • Are you running pipelines or freestyle or some other job type?
  • Are you able to share your pipeline at all? At least the statement(s) that cause the error?
  • I’m assuming based on the dockerjava.api stuff, your using tcp connection, did you configure that at all, or is that just something windows does out of the box?
  • Is the docker image your using public? Can we take a look? Or if its private, what is the FROM based on?

I vaguely remember a while back on irc investigating and finding isUnix() function didnt properly detect windows, but the person left and didn’t come back so we never got enough info to make a report.

  1. We are using Windows Server 2016 Integrated Docker containers.
  2. I have created a custom base docker image for windows server 2016 based upon the publically available agent images on github.
  3. We are using a tcp connection yes, utilizing client certs. I am able to connect from a local machine and the “test connection” functionality in the Clouds returns “success”.
  4. The pipeline is just a simple hello world.


pipeline {
agent {
label ‘windows’

stages {
    stage('Hello') {
        steps {
            echo 'Hello World'


Console Log:

Started by user [Jonathon Lamon (Admin)]
Running as [Jonathon Lamon (Admin)]
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘[windows]’ are offline

Error on template:

Note: Disabled 1 min 58 sec ago due to error. Will re-enable in 3 min 1 sec.
Reason: Template provisioning failed.

com.github.dockerjava.api.exception.BadRequestException: {“message”:“container fb73c60dd664a5af19e7976d3696625c42ccd2089bbc9d33dd8c46c551f75012 encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {“CommandLine”:”/bin/sh",“User”:“jenkins”,“WorkingDirectory”:“C:\\Users\\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Current\\Bin\\Roslyn”},“EmulateConsole”:true,“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“ConsoleSize”:[0,0]}"} at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0( at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0( at at at at at io.netty.handler.logging.LoggingHandler.channelRead( at at at at io.netty.handler.codec.MessageToMessageDecoder.channelRead( at at at at$DelegatingChannelHandlerContext.fireChannelRead( at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead( at io.netty.handler.codec.ByteToMessageDecoder.channelRead( at at at at at io.netty.handler.timeout.IdleStateHandler.channelRead( at at at at io.netty.handler.ssl.SslHandler.unwrap( at io.netty.handler.ssl.SslHandler.decodeJdkCompatible( at io.netty.handler.ssl.SslHandler.decode( at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection( at io.netty.handler.codec.ByteToMessageDecoder.callDecode( at io.netty.handler.codec.ByteToMessageDecoder.channelRead( at at at at$HeadContext.channelRead( at at at at$ at at at at at io.netty.util.concurrent.SingleThreadEventExecutor$ at at

Base Image:


#Base windows image with .net and jdk 8
FROM mcr[dot]microsoft[dot]com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2016
SHELL [“powershell”, “-Command”, “$ErrorActionPreference = ‘Stop’; $ProgressPreference = ‘SilentlyContinue’;”]

#install jdk and add to environment

RUN Invoke-WebRequest “[http://]repos[dot]azul[dot]com/azure-only/zulu/packages/zulu-8/8u292/zulu-8-azure-jdk_8.54.0.21-8.0.292-win_x64.msi” -OutFile “$env:TEMP\azuld-jdk.msi” -UseBasicParsing ; Start-Process msiexec.exe -Wait -NoNewWindow -PassThru -ArgumentList "/i","$env:TEMP\azuld-jdk.msi","/qn","APPLICATIONROOTDIRECTORY=C:\PROGRA~1";
Remove-Item “$env:TEMP\azuld-jdk.msi” -Force;

RUN setx /M JRE_HOME ‘C:\Program Files\zulu-8\jre’
RUN setx /M JAVA_HOME ‘C:\Program Files\zulu-8’

#install git for windows

RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; $url = $('[https://]github[dot]com/git-for-windows/git/releases/download/v{0}.windows.{1}/MinGit-{0}' -f $env:GIT_VERSION, $env:GIT_PATCH_VERSION) ;
Write-Host “Retrieving $url…” ; Invoke-WebRequest $url -OutFile '' -UseBasicParsing ;
Expand-Archive -DestinationPath c:\mingit ; `
Remove-Item -Force

RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; $url = $('[https://]github[dot]com/git-lfs/git-lfs/releases/download/v{0}/git-lfs-windows-amd64-v{0}.zip' -f $env:GIT_LFS_VERSION) ;
Write-Host “Retrieving $url…” ; Invoke-WebRequest $url -OutFile '' -UseBasicParsing ;
Expand-Archive -DestinationPath c:\mingit\mingw64\bin ; Remove-Item -Force ;
& C:\mingit\cmd\git.exe lfs install ; $CurrentPath = (Get-Itemproperty -path 'hklm:\system\currentcontrolset\control\session manager\environment' -Name Path).Path ;
$NewPath = $CurrentPath + ‘;C:\mingit\cmd’ ; `
Set-ItemProperty -path ‘hklm:\system\currentcontrolset\control\session manager\environment’ -Name Path -Value $NewPath

Agent Image:


#Base jenkins agent with .net 4.8, jdk8, and jenkins agent
#based upon images @ [https://]github[dot]com/jenkinsci/docker-agent
FROM /windows:windowsservercore-net48-zulujdk8

LABEL Description=“This is a base image, which provides the Jenkins agent executable (agent.jar)” Vendor=“Jenkins project” Version="${VERSION}"

#install jenkins agent
ARG user=jenkins


RUN net user “$env:user” /add /expire:never /passwordreq:no ; net localgroup Administrators /add $env:user ;
Set-LocalUser -Name $env:user -PasswordNeverExpires 1; `
New-Item -ItemType Directory -Path C:/ProgramData/Jenkins | Out-Null

ARG AGENT_ROOT=C:/Users/$user


Get the Agent from the Jenkins Artifacts Repository

RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-WebRequest $('[https://]repo[dot]jenkins-ci[dot]org/public/org/jenkins-ci/main/remoting/{0}/remoting-{0}.jar' -f $env:VERSION) -OutFile $(Join-Path C:/ProgramData/Jenkins $env:AGENT_FILENAME) -UseBasicParsing ;
Invoke-WebRequest $(’[https://]repo[dot]jenkins-ci[dot]org/public/org/jenkins-ci/main/remoting/{0}/remoting-{0}.jar.sha1’ -f $env:VERSION) -OutFile (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME) -UseBasicParsing ; if ((Get-FileHash (Join-Path C:/ProgramData/Jenkins $env:AGENT_FILENAME) -Algorithm SHA1).Hash -ne (Get-Content (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME))) {exit 1} ;
Remove-Item -Force (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME)

USER $user

RUN New-Item -Type Directory $(’{0}/.jenkins’ -f $env:AGENT_ROOT) | Out-Null ; `
New-Item -Type Directory $env:AGENT_WORKDIR | Out-Null


Oh this is great information…

ah, your using docker templated agents, not using docker agents inside of pipelines. I know less about that than I do windows agents.

Based on docker-plugin/ at a33c764feda6a6ad7c4cf709c04c966610ceab5f · jenkinsci/docker-plugin · GitHub I’m guessing for your templates your not providing your own command? I don’t know what the equiv is in windows, cmd or pwsh or something?

The only other mention is ssh keys, and i don’t think that applies - docker-plugin/ at fd1ea7ded448f7767ae33cef5d9683fb977a1175 · jenkinsci/docker-plugin · GitHub

That is kind of what I came here to find out because of the error. IE if I could actually provision my own command and asked about the configuration settings. I do not see where in the UI I would tell it to use something other than /bin/sh. Is it what I would configure below?

Would that even work? Seems odd it would be configuration of the node template and not at the node level itself.

Apparently that is the correct thing to set.

I set “Docker Command” to “cmd”. Then received an error because I had the root file system set to C:/Users/jenkins/Work with an error that it didn’t exist in the container. Changed that to C:/Users/jenkins and now I received the following error:

Connecting to docker container 75cb203f471b55bea89fac91dd7aa616b8c93fa8bb0efe93a049da2b5817f3fb, running command java -jar C:/Users/jenkins/remoting-4.7.jar -noReconnect -noKeepAlive -agentLog C:/Users/jenkins/agent.log
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
Api-Version: 1.41
Docker-Experimental: false
Ostype: windows
Server: Docker/20.10.5 (windows)
container 75cb203f471b55bea89fac91dd7aa616b8c93fa8bb0efe93a049da2b5817f3fb encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The requested compute system operation is not valid in the current state. (0xc0370105) extra info: {“CommandLine”:“java -jar C:/Users/jenkins/remoting-4.7.jar -noReconnect -noKeepAlive -agentLog C:/Users/jenkins/agent.log”,“User”:“jenkins”,“WorkingDirectory”:“C:\Users\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Roslyn”},“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“CreateStdErrPipe”:true,“ConsoleSize”:[0,0]}
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins

Hello @jonl_percsol , could you provide a minimalistic reproduction case (such as a docker-compose that I could use on my Windows machine with Docker, to start a Jenkins controller with the correct setup) please? It’s really hard to guess what is going wrong without having the configuration.

The error you seem to have seems related to seomthing happening while the container is starting. There are different way of debugging it, like you would with a normal Linux Docker container (or Kubernetes pods):

  • If you can disable the “Agent deletion”, you can check with docker ps -a on the Windows Server with the Docker Engine that you have the failed agent. Then, get the logs of the agent with docker logs <container id> to get a glimpse of what happened.
  • Some requirement for an inbound agent might be missing in your setup (can the container route back to the controller, or can the controller route to the agent if using SSH? ). With the “HTTP/101 upgraded”, I bet that websocket might be the culprit: could be worth it to try without websockets).
    You might want to create a “static agent” of type “inbound/jnlp” and spawn manually an interactive windows container, basd on your image, to run the full java command that is expected to start the agent (the whole java -jar <...> command shown on the Jenkins UI) : any error here (that would certainly mathc what you see on your failed container logs) would help to indicate what is going wrong.

To answer what we (the jenkins-infra) do with Windows containers:

  • Either we have Windows Server (with Docker) agents dynamically provisionned on Azure or EC2 to provide a VM with a Docker Engine within
  • Either we use Azure ACI service to spawn containers (Kubernetes incoming as alternative).
    => we do NOT use the “docker-plugin” as we need dynamic capability which is not provided by a single machine with Docker, so the context is different than yours as I udnerstand it.

We are running a Windows Server 2016 with Docker. On a separate linux server I am running Jenkins. I do not know of a way that I would be able to recreate a minimal test case that could be easily deployed. What I have done is added the Windows Server Docker instance as a cloud connecting over tcp. It is the configuration of this cloud that I am attempting to get right. Below is my current configuration:



Name: docker-win-build
Docker Host URI: tcp://<ip/hostname>:2376
Server Credentials:
Container Cap: 100

Docker Agent Templates:
Labels: windows
Name: Windows Server Core 1809 with JDK8
Docker Image: /jenkins/agents/jdk8-windowservercore-2016:latest (this is the agent image in my comment above)
Docker Command:cmd
Remote File System Root: C:/Users/jenkins
Usage: Only build jobs with label expressions matching this node
Connect method: Attach Docker container
User: jenkins