Setting up windows cloud agents

Speaking of windows images, are there any good instructions on setting up a Windows Server based Docker node in the Manage Nodes and Clouds / Configure Clouds section? I have setup a windows server for running windows based jenkins nodes/agents and I am having a hard time finding all the proper configuration settings.

I split this off of Java 11 as default in Docker images - #2 as its not really about the change to java 11 inside agent images.

I am having a hard time finding all the proper configuration settings.

What do you mean by proper? Is something not working that you want fixed?

Maybe @dduportal can share what they did for jenkins infra and windows vms (though I think its mostly via azure)

There appears to be zero documentation on the setup of windows agents that I can find and when I attempt to run a build on one I receive the error bellow, which indicates that it is attempting to run /bin/sh, which of course does not exist on windows. There has to be someone somewhere has to be using the feature, yes? However, I can’t find any examples of actually configuring a windows docker node within Jenkins, so I am not sure if the error is because of a configuration issue, a bug, or a combination of both.

Error:

com.github.dockerjava.api.exception.BadRequestException: {“message”:“container a1966484e2336cbb68018153e85cfb938e5dd7d20f4a3b0725ab96e3a18f8153 encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {“CommandLine”:”/bin/sh",“User”:“jenkins”,“WorkingDirectory”:“C:\\Users\\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Current\\Bin\\Roslyn”},“EmulateConsole”:true,“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“ConsoleSize”:[0,0]}"}

Even outside of jenkins, running real windows docker images inside of windows is fairly new. For a long time windows would run a linux vm and run docker on that. That doesn’t excuse the lack of docs, but between that and people being volunteer it might explain it a bit.

I’m guessing we’ll end up creating a bug for it. I’m by no means an expert, but I can help narrow it down for those who are. For now more information is needed.

  • Are you running pipelines or freestyle or some other job type?
  • Are you able to share your pipeline at all? At least the statement(s) that cause the error?
  • I’m assuming based on the dockerjava.api stuff, your using tcp connection, did you configure that at all, or is that just something windows does out of the box?
  • Is the docker image your using public? Can we take a look? Or if its private, what is the FROM based on?

I vaguely remember a while back on irc investigating and finding isUnix() function didnt properly detect windows, but the person left and didn’t come back so we never got enough info to make a report.

  1. We are using Windows Server 2016 Integrated Docker containers.
  2. I have created a custom base docker image for windows server 2016 based upon the publically available agent images on github.
  3. We are using a tcp connection yes, utilizing client certs. I am able to connect from a local machine and the “test connection” functionality in the Clouds returns “success”.
  4. The pipeline is just a simple hello world.

Pipeline:

pipeline {
agent {
label ‘windows’
}

stages {
    stage('Hello') {
        steps {
            echo 'Hello World'
        }
    }
}

}

Console Log:

Started by user [Jonathon Lamon (Admin)]
Running as [Jonathon Lamon (Admin)]
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘[windows]’ are offline

Error on template:

Note: Disabled 1 min 58 sec ago due to error. Will re-enable in 3 min 1 sec.
Reason: Template provisioning failed.

com.github.dockerjava.api.exception.BadRequestException: {“message”:“container fb73c60dd664a5af19e7976d3696625c42ccd2089bbc9d33dd8c46c551f75012 encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The system cannot find the file specified. (0x2) extra info: {“CommandLine”:”/bin/sh",“User”:“jenkins”,“WorkingDirectory”:“C:\\Users\\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\MSBuild\\Current\\Bin\\Roslyn”},“EmulateConsole”:true,“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“ConsoleSize”:[0,0]}"} at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:99) at com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1432) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1199) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:648) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:583) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748)

Base Image:

escape=`

#Base windows image with .net and jdk 8
FROM mcr[dot]microsoft[dot]com/dotnet/framework/sdk:4.8-windowsservercore-ltsc2016
SHELL [“powershell”, “-Command”, “$ErrorActionPreference = ‘Stop’; $ProgressPreference = ‘SilentlyContinue’;”]

#install jdk and add to environment

RUN Invoke-WebRequest “[http://]repos[dot]azul[dot]com/azure-only/zulu/packages/zulu-8/8u292/zulu-8-azure-jdk_8.54.0.21-8.0.292-win_x64.msi” -OutFile “$env:TEMP\azuld-jdk.msi” -UseBasicParsing ; Start-Process msiexec.exe -Wait -NoNewWindow -PassThru -ArgumentList "/i","$env:TEMP\azuld-jdk.msi","/qn","APPLICATIONROOTDIRECTORY=C:\PROGRA~1";
Remove-Item “$env:TEMP\azuld-jdk.msi” -Force;

RUN setx /M JRE_HOME ‘C:\Program Files\zulu-8\jre’
RUN setx /M JAVA_HOME ‘C:\Program Files\zulu-8’

#install git for windows

ARG GIT_VERSION=2.31.0
ARG GIT_PATCH_VERSION=1
RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; $url = $('[https://]github[dot]com/git-for-windows/git/releases/download/v{0}.windows.{1}/MinGit-{0}-64-bit.zip' -f $env:GIT_VERSION, $env:GIT_PATCH_VERSION) ;
Write-Host “Retrieving $url…” ; Invoke-WebRequest $url -OutFile 'mingit.zip' -UseBasicParsing ;
Expand-Archive mingit.zip -DestinationPath c:\mingit ; `
Remove-Item mingit.zip -Force

ARG GIT_LFS_VERSION=2.13.2
RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; $url = $('[https://]github[dot]com/git-lfs/git-lfs/releases/download/v{0}/git-lfs-windows-amd64-v{0}.zip' -f $env:GIT_LFS_VERSION) ;
Write-Host “Retrieving $url…” ; Invoke-WebRequest $url -OutFile 'GitLfs.zip' -UseBasicParsing ;
Expand-Archive GitLfs.zip -DestinationPath c:\mingit\mingw64\bin ; Remove-Item GitLfs.zip -Force ;
& C:\mingit\cmd\git.exe lfs install ; $CurrentPath = (Get-Itemproperty -path 'hklm:\system\currentcontrolset\control\session manager\environment' -Name Path).Path ;
$NewPath = $CurrentPath + ‘;C:\mingit\cmd’ ; `
Set-ItemProperty -path ‘hklm:\system\currentcontrolset\control\session manager\environment’ -Name Path -Value $NewPath

Agent Image:

escape=`

#Base jenkins agent with .net 4.8, jdk8, and jenkins agent
#based upon images @ [https://]github[dot]com/jenkinsci/docker-agent
FROM /windows:windowsservercore-net48-zulujdk8

ARG VERSION=4.7
LABEL Description=“This is a base image, which provides the Jenkins agent executable (agent.jar)” Vendor=“Jenkins project” Version="${VERSION}"

#install jenkins agent
ARG user=jenkins

ARG AGENT_FILENAME=agent.jar
ARG AGENT_HASH_FILENAME=$AGENT_FILENAME.sha1

RUN net user “$env:user” /add /expire:never /passwordreq:no ; net localgroup Administrators /add $env:user ;
Set-LocalUser -Name $env:user -PasswordNeverExpires 1; `
New-Item -ItemType Directory -Path C:/ProgramData/Jenkins | Out-Null

ARG AGENT_ROOT=C:/Users/$user
ARG AGENT_WORKDIR=${AGENT_ROOT}/Work

ENV AGENT_WORKDIR=${AGENT_WORKDIR}

Get the Agent from the Jenkins Artifacts Repository

RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; Invoke-WebRequest $('[https://]repo[dot]jenkins-ci[dot]org/public/org/jenkins-ci/main/remoting/{0}/remoting-{0}.jar' -f $env:VERSION) -OutFile $(Join-Path C:/ProgramData/Jenkins $env:AGENT_FILENAME) -UseBasicParsing ;
Invoke-WebRequest $(’[https://]repo[dot]jenkins-ci[dot]org/public/org/jenkins-ci/main/remoting/{0}/remoting-{0}.jar.sha1’ -f $env:VERSION) -OutFile (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME) -UseBasicParsing ; if ((Get-FileHash (Join-Path C:/ProgramData/Jenkins $env:AGENT_FILENAME) -Algorithm SHA1).Hash -ne (Get-Content (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME))) {exit 1} ;
Remove-Item -Force (Join-Path C:/ProgramData/Jenkins $env:AGENT_HASH_FILENAME)

USER $user

RUN New-Item -Type Directory $(’{0}/.jenkins’ -f $env:AGENT_ROOT) | Out-Null ; `
New-Item -Type Directory $env:AGENT_WORKDIR | Out-Null

VOLUME ${AGENT_ROOT}/.jenkins
VOLUME ${AGENT_WORKDIR}
WORKDIR ${AGENT_ROOT}

Oh this is great information…

ah, your using docker templated agents, not using docker agents inside of pipelines. I know less about that than I do windows agents.

Based on docker-plugin/DockerComputerConnector.java at a33c764feda6a6ad7c4cf709c04c966610ceab5f · jenkinsci/docker-plugin · GitHub I’m guessing for your templates your not providing your own command? I don’t know what the equiv is in windows, cmd or pwsh or something?

The only other mention is ssh keys, and i don’t think that applies - docker-plugin/DockerComputerSSHConnector.java at fd1ea7ded448f7767ae33cef5d9683fb977a1175 · jenkinsci/docker-plugin · GitHub

That is kind of what I came here to find out because of the error. IE if I could actually provision my own command and asked about the configuration settings. I do not see where in the UI I would tell it to use something other than /bin/sh. Is it what I would configure below?

Would that even work? Seems odd it would be configuration of the node template and not at the node level itself.

Apparently that is the correct thing to set.

I set “Docker Command” to “cmd”. Then received an error because I had the root file system set to C:/Users/jenkins/Work with an error that it didn’t exist in the container. Changed that to C:/Users/jenkins and now I received the following error:

Connecting to docker container 75cb203f471b55bea89fac91dd7aa616b8c93fa8bb0efe93a049da2b5817f3fb, running command java -jar C:/Users/jenkins/remoting-4.7.jar -noReconnect -noKeepAlive -agentLog C:/Users/jenkins/agent.log
HTTP/1.1 101 UPGRADED
Content-Type: application/vnd.docker.raw-stream
Connection: Upgrade
Upgrade: tcp
Api-Version: 1.41
Docker-Experimental: false
Ostype: windows
Server: Docker/20.10.5 (windows)
container 75cb203f471b55bea89fac91dd7aa616b8c93fa8bb0efe93a049da2b5817f3fb encountered an error during hcsshim::System::CreateProcess: failure in a Windows system call: The requested compute system operation is not valid in the current state. (0xc0370105) extra info: {“CommandLine”:“java -jar C:/Users/jenkins/remoting-4.7.jar -noReconnect -noKeepAlive -agentLog C:/Users/jenkins/agent.log”,“User”:“jenkins”,“WorkingDirectory”:“C:\Users\jenkins”,“Environment”:{“AGENT_WORKDIR”:“C:/Users/jenkins/Work”,“COMPLUS_NGenProtectedProcess_FeatureEnabled”:“0”,“NUGET_VERSION”:“5.9.1”,“ROSLYN_COMPILER_LOCATION”:“C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Current\Bin\Roslyn”},“CreateStdInPipe”:true,“CreateStdOutPipe”:true,“CreateStdErrPipe”:true,“ConsoleSize”:[0,0]}
ERROR: Unexpected error in launching an agent. This is probably a bug in Jenkins