Hello all,
We have Jenkins running on an EC2 Ubuntu instance on AWS, sitting behind an nginx ingress proxy, which is the target group of an Application Load Balancer (ALB). The ALB is handling TLS and also doing SSO so that requests are authenticated and decrypted before being forwarded as http (not encrypted) to the nginx reverse proxy, which forwards the requests to the Jenkins Controller. All is well, for this - users can go to the externally-facing IP address of the ALB and get through to Jenkins.
The problem comes when we want to use a Windows-based Jenkins Node on site. We would like this to connect to the Jenkins Controller, however we have a catch-22 situation - the Jenkins Controller agent doesn’t understand SSO and so fails the port-forwarding at the ALB. So I configured an additional rule on the ALB to forward incoming requests on port 8080 (having handled TLS) to the same port 8080 directly on the Jenkins Controller instance, thus bypassing the nginx proxy.
This worked ok, in that I can now curl the agent.jar file by accessing it on port 8080. However the returned jar file, when executed with the script given on a new remote node section, fails because the system address is set to be on port 443 (for regular users) not 8080 (for skipping SSO). I’ve overridden the jnlp request to go back to port 8080, and now we get the jnlp file. Great!
Except that the jnlp file seems to cause agent.jar to go and try and connect on port 443 again (not 8080). So I configured Jenkins to instead listen for agents connecting on a different port (8081), and added another ALB rule to forward 8081 (after handling TLS).
The problem is that despite configuring my remote node to be “tunnelled” through port 8081, and configuring the TCP port for incoming agents to be 8081, the jnlp seems to be ignoring this and I get the following log:
C:\temp>jdk-17\bin\java -jar agent.jar -jnlpUrl https://<aws_alb_address>:8080/jenkins/computer/test-node/jenkins-agent.jnlp -secret c30ab8...554b -workDir "C:\temp\jenkins"
Feb 29, 2024 2:50:28 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using C:\temp\jenkins\remoting as a remoting work directory
Feb 29, 2024 2:50:28 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to C:\temp\jenkins\remoting
Feb 29, 2024 2:50:29 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: test-node
Feb 29, 2024 2:50:29 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3160.vd76b_9ddd10cc
Feb 29, 2024 2:50:29 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using C:\temp\jenkins\remoting as a remoting work directory
Feb 29, 2024 2:50:29 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://<aws_alb_address>]
Feb 29, 2024 2:50:29 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Could not locate server among [https://<aws_alb_address>/jenkins/]; waiting 10 seconds before retry
java.io.IOException: https://<aws_alb_address>/jenkins/tcpSlaveAgentListener/ appears to be publishing an invalid X-Instance-Identity.
at org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.resolve(JnlpAgentEndpointResolver.java:277)
at hudson.remoting.Engine.innerRun(Engine.java:761)
at hudson.remoting.Engine.run(Engine.java:543)
It looks to me like Jenkins is completely ignoring my setting for the listening port.
If I abandon the use of port 8081 as the listener, and instead try changing my Jenkins URL to be https://<aws_alb_address>:8080/jenkins and use websockets to connect, it goes straight through. However links generated for users to use (and uploaded to Bitbucket as build statuses) are now broken as they include :8080.
Does anyone have any suggestions? I believe this should work using the tunnelling mode (unless I’ve misunderstood it) - the Jenkins Controller seems to be completely ignoring the node configuration for accessing things via the specified tunnel.
Many thanks for any pointers…
Richard
Jenkins setup:
Jenkins: 2.426.3
OS: Linux - 6.2.0-1018-aws
Java: 17.0.10 - Private Build (OpenJDK 64-Bit Server VM)
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
artifact-manager-s3:845.vb_0805c11eb_26
authentication-tokens:1.53.v1c90fd9191a_b_
aws-credentials:218.v1b_e9466ec5da_
aws-global-configuration:130.v35b_7b_96f53c3
aws-java-sdk-ec2:1.12.633-430.vf9a_e567a_244f
aws-java-sdk-minimal:1.12.633-430.vf9a_e567a_244f
badge:1.9.1
basic-branch-build-strategies:81.v05e333931c7d
bitbucket:241.v6d24a_57f9359
bootstrap5-api:5.3.2-3
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1148.vce12cfcdf090
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-bitbucket-branch-source:873.v0ed259216f8d
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-95.v22a_d30ee5d36
copyartifact:722.v0662a_9b_e22a_c
credentials:1319.v7eb_51b_3a_c97b_
credentials-binding:657.v2b_19db_7d6e6d
dark-theme:416.v535839b_c4e88
data-tables-api:1.13.8-2
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-workflow:572.v950f58993843
durable-task:550.v0930093c4b_a_6
ec2:1648.vf3d852e00486
echarts-api:5.4.3-2
email-ext:2.104
font-awesome-api:6.5.1-2
git:5.2.1
git-client:4.6.0
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1772.va_69eda_d018d4
global-slack-notifier:1.5
gradle:2.10
groovy-postbuild:228.vcdb_cf7265066
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.32
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.16.1-373.ve709c6871598
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jjwt-api:0.11.5-77.v646c772fddb_0
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-1
jsch:0.2.16-86.v42e010d9484b_
json-path-api:2.9.0-33.v2527142f2e1d
junit:1259.v65ffcef24a_88
jwt-auth:0.3.0
ldap:711.vb_d1a_491714dc
lockable-resources:1232.v512d6c434eb_d
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
mina-sshd-api-common:2.12.0-90.v9f7fb_9fa_3d3b_
mina-sshd-api-core:2.12.0-90.v9f7fb_9fa_3d3b_
node-iterator-api:55.v3b_77d4032326
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2175.v76a_fff0a_2618
pipeline-model-definition:2.2175.v76a_fff0a_2618
pipeline-model-extensions:2.2175.v76a_fff0a_2618
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2175.v76a_fff0a_2618
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.8.0
resource-disposer:0.23
scm-api:683.vb_16722fb_b_80b_
script-security:1326.vdb_c154de8669
skip-notifications-trait:313.vd1337c8f8134
slack:684.v833089650554
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.1
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
theme-manager:215.vc1ff18d67920
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.133.vfb_8a_7b_9c5dd1
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3853.vb_a_490d892963
workflow-durable-task-step:1331.vc8c2fed35334
workflow-job:1385.vb_58b_86ea_fff1
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:415.v434365564324
workflow-step-api:657.v03b_e8115821b_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45