Seeing issue with pod connection after Kubernetes Plugin Upgrade

Hey Team,
Service Details::
Jenkins Core Version: 2.426.3
Java: 11.0.16.1
Kubernetes Plugin: 4186.v1d804571d5d4

After moving kubernetes from 1.30.11 to 4186.v1d804571d5d4, I am seeing an issue when Jenkins tries to create a Agent.
Test Connection ::

Error from the Catalina Logs on Jenkins

12-Jun-2024 04:55:07.936 INFO [TCP agent connection handler #35 with /100.101.128.101:26044] hudson.TcpSlaveAgentListener$ConnectionHandler.run Connection #35 from /100.101.128.101:26044 failed: null

I tried to manually launch the agent on a Unix VM; this is the error seen

[root@cne-jenkins bin]# /scratch/sdaas/runtime/jdk-11.0.16.1/bin/java -jar agent.jar -jnlpUrl https://<hostname>:8484/jenkins/computer/oscs%2Doci%2Dscan%2Dksnv5%2D3kbtc/jenkins-agent.jnlp -secret xxxxxxx -workDir ""
Jun 12, 2024 4:56:35 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /remoting as a remoting work directory
Jun 12, 2024 4:56:35 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /remoting
Failed to obtain https://<hostname>:8484/jenkins/computer/oscs%2Doci%2Dscan%2Dksnv5%2D3kbtc/jenkins-agent.jnlp?encrypt=true
java.io.IOException: Failed to load https://<hostname>:8484/jenkins/computer/oscs%2Doci%2Dscan%2Dksnv5%2D3kbtc/jenkins-agent.jnlp?encrypt=true: 404 null
	at hudson.remoting.Launcher.parseJnlpArguments(Launcher.java:514)
	at hudson.remoting.Launcher.run(Launcher.java:346)
	at hudson.remoting.Launcher.main(Launcher.java:297)
Waiting 10 seconds before retry

:question: :thought_balloon: Can someone please assist me with what the issue can be with. FYI, There are no concerns with kubernetes v1.30.11.

Also, :question: Can WebSocket be supported with Tomcat 9? Currently I am seeing the below warning


But, the documentation does say javax.servlet is supported with TomCat 9

Lastly, Apart from setting the JNLP4-connect protocol under Manage Jenkins → Security → Agents → Protocols, Is there a JAVA_OPTS or env variable that can impose Jenkins to use JNLP4-connect always.

Lately, we have been seeing issue while getting the Agent Protocol even though the Protocol is enabled under Security.

May 31, 2024 7:02:45 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
May 31, 2024 7:02:45 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
May 31, 2024 7:02:55 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
	at hudson.remoting.Engine.innerRun(Engine.java:778)
	at hudson.remoting.Engine.run(Engine.java:540)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
	at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
	at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164)
	at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.lambda$start$0(AckFilterLayer.java:177)
	at org.jenkinsci.remoting.protocol.IOHub$DelayedRunnable.run(IOHub.java:959)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
	at java.base/java.lang.Thread.run(Thread.java:829)

May 31, 2024 7:02:56 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
	at hudson.remoting.Engine.onConnectionRejected(Engine.java:864)
	at hudson.remoting.Engine.innerRun(Engine.java:804)
	at hudson.remoting.Engine.run(Engine.java:540)

Hi,

The error messages you’re seeing suggest a few different issues :cry: :

  1. Failed to load jenkins-agent.jnlp: 404 null: it looks like the Jenkins agent is unable to fetch the JNLP file from the Jenkins controller. This could be due to a number of reasons, such as network connectivity issues, incorrect Jenkins URL, or the agent not being properly configured on the Jenkins controller. It would help if you verified that the Jenkins URL is correct and that the agent is properly configured on the Jenkins controller.
  2. Connection closed before acknowledgment sent: the connection between the Jenkins controller and the agent may be closed before the connection can be fully established. This could be due to network issues, or it could be a problem with the JNLP4-connect protocol. You might want to try using a different protocol to see if that resolves the issue.
  3. WebSocket support with Tomcat 9: Yes, WebSocket is supported with Tomcat 9. However, the warning message you’re seeing suggests that the WebSocket feature is not enabled in your Tomcat configuration. You should check your Tomcat configuration to ensure that WebSocket is enabled.
  4. Setting the JNLP4-connect protocol: I think you can set the JNLP_PROTOCOL_OPTS environment variable to specify the protocols that the Jenkins agent should use. For example, you could set JNLP_PROTOCOL_OPTS=-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true -Dorg.jenkinsci.remoting.engine.JnlpProtocol4.disabled=false to disable the JNLP3 protocol and enable the JNLP4 protocol.
1 Like

Thank you @poddingue I will work the recommendations :star:

Sometimes, we also see

13:01:45  INFO: Waiting for ProtocolStack to start.
13:01:45  Jun 13, 2024 1:01:17 PM org.jenkinsci.remoting.protocol.impl.AckFilterLayer abort
13:01:45  WARNING: [JNLP4-connect connection to cne-jenkins-drtest.us.oracle.com/100.101.128.226:50000] Incorrect acknowledgement sequence, expected 0x000341434b got 0x4469736162
13:01:45  Jun 13, 2024 1:01:17 PM hudson.remoting.jnlp.Main$CuiListener status
13:01:45  INFO: Protocol JNLP4-connect encountered an unexpected exception
13:01:45  java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
13:01:45  	at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
13:01:45  	at hudson.remoting.Engine.innerRun(Engine.java:778)
13:01:45  	at hudson.remoting.Engine.run(Engine.java:540)
13:01:45  Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent

:question: How can we control the acknowledgement sequence?