# Can JNLP Protocol get auto-disabled due to JVM memory allocated **URL:** \ **Category:** Ask a question\ **Created:** [May 5, 2024, 5:35pm UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831 "2024-05-05T17:35:34Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hpriya](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hpriya](https://community.jenkins.io/u/hpriya)\ **Post date:** [May 5, 2024, 5:35pm UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/1 "2024-05-05T17:35:34Z")
Hey Experts, We have **Jenkins Core** : 2.426.3 **Java** : 11.0.16.1 Our current JAVA\_OPTS is set to ```auto export JAVA_OPTS="-DJENKINS_HOME=$JENKINS_HOME \ -XX:ReservedCodeCacheSize=512m -XX:+UseCodeCacheFlushing -Xms$MIN_HEAP_SIZE -Xmx$MAX_HEAP_SIZE \ -XX:+UseG1GC -XX:G1ReservePercent=20 \ -Xloggc:/usr/local/tomcat/logs/jenkins.gc-%t.log -XX:+UseGCLogFileRotation \ -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=2M \ -XX:+IgnoreUnrecognizedVMOptions \ -XX:-PrintGCDetails -XX:+PrintGCDateStamps -XX:-PrintTenuringDistribution \ -Dhudson.ClassicPluginStrategy.useAntClassLoader=true \ -Dkubernetes.websocket.ping.interval=20000 \ -Dhudson.slaves.SlaveComputer.allowUnsupportedRemotingVersions=true \ -Djava.awt.headless=true -Dhudson.slaves.ChannelPinger.pingIntervalSeconds=300" export JAVA_OPTS="$JAVA_OPTS -Dhudson.model.DirectoryBrowserSupport.CSP=\"default-src 'none' netdna.bootstrapcdn.com; img-src 'self' 'unsafe-inline' data:; style-src 'self' 'unsafe-inline' https://www.google.com ajax.googleapis.com netdna.bootstrapcdn.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://www.google.com ajax.googleapis.com netdna.bootstrapcdn.com cdnjs.cloudflare.com; child-src 'self';\" -Dcom.cloudbees.hudson.plugins.folder.computed.ThrottleComputationQueueTaskDispatcher.LIMIT=100" ``` where MIN\_HEAP\_SIZE = 12288m & MAX\_HEAP\_SIZE = 200G. We are seeing > java.lang.Exception: The server rejected the connection: None of the protocols were accepted even though the _Mange Jenkins → Security → Agent Protocol → Inbound TCP Agent Protocol/4 TLS Encryption_ is enabled and intact. Also, we are seeing the port **50000** toggle on our monitoring dashboard when the event occurs. (We suspect that the JVM’s constrained memory usage is responsible for this behavior, leading to the 50000 port being pulled down and resulting in JNLP connection errors.) We are seeking assistance on the following: 1. The correlation between JNLP and JVM memory usage. 2. Whether the JVM prioritizes cached memory over actual memory usage. 3. Any additional factors that could influence port 50000 for JNLP connections in Jenkins. Regards Hema ---
**Author:** ![hpriya](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hpriya](https://community.jenkins.io/u/hpriya)\ **Post date:** [May 5, 2024, 5:43pm UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/2 "2024-05-05T17:43:53Z")
Agent Protocol Setting :: ![agent-jenkins-jnlp-port](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/f/fab1e8bb6485734f06f6d11833dc3a6115ec8b39.png) JVM memory used :: ![jvm-used](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/7/796fedb873effc8f06249f27a0dd0c2a3154693d.png) JVM memory cached :: ![jvm-cached](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/7/754822d33ed6de8c359003c067234420e41dcb87.png) JNLP Port 50000 fluctuations :: ![jnlp-50000](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/d/d2c6a2f8eea35205e40ddbca13bbe886aa6cf0c4.png) ---
**Author:** ![poddingue](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/poddingue/32/985_2.png) [@poddingue](https://community.jenkins.io/u/poddingue)\ **Post date:** [May 6, 2024, 7:56am UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/3 "2024-05-06T07:56:38Z")
Hi @hpriya, I’m not a JVM expert by any means, but given the large heap sizes you’re using (`-Xms12288m -Xmx200G`), the JVM may be struggling to manage such a large heap, especially if the physical memory on the machine is not significantly larger than the maximum heap size. You might want to consider reducing the maximum heap size to see if that alleviates the problem. You could also consider enabling more detailed GC logging to understand the JVM’s memory management behavior better. It’s also possible that the Jenkins controller is running out of system resources (such as file descriptors or threads), which could cause it to close connections. 🤷 ---
**Author:** ![hpriya](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hpriya](https://community.jenkins.io/u/hpriya)\ **Post date:** [May 6, 2024, 8:51am UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/4 "2024-05-06T08:51:34Z")
> physical memory on the machine We are using a VM with ocpus=80, Memory=1280GB. It’s a OL8 VM. That should be sufficient to set a heap-size of 200G. > You could also consider enabling more detailed GC logging to understand the JVM’s memory management behavior better. Okay. > It’s also possible that the Jenkins controller is running out of system resources (such as file descriptors or threads) W.r.t threads; the max threads on the server and jenkins service deployed is 10315681 ```auto [root@container-name tomcat]# cat /proc/sys/kernel/threads-max 10315681 [root@hostname logs]# cat /proc/sys/kernel/threads-max 10315681 ``` We did see the error below; ```auto 12-Apr-2024 15:15:38.636 SEVERE [TCP agent listener port=50000] hudson.TcpSlaveAgentListener.run Failed to accept TCP connections java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached ``` But the memory usage is less than the max value. We have the max-limit set to 200G, our monitoring board shows the usage was ~110G. Any input on which memory exhausted/ Jenkins is complaining about will be helpful : ) This is the ulimit on the OL8 server ```auto [root@hostname]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 5157840 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 5157840 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ``` This is the ulimit config inside the Jenkins containers deployed on the server ```auto [sdaasbld@container-name ~]$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 5157840 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 8192 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 8192 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited ``` We manage the /etc/security/limits.conf file with the below config setup ```auto sdaasbld soft nofile 8192 sdaasbld hard nofile 65536 sdaasbld soft nproc 8192 sdaasbld hard nproc 16384 ``` Probable suspect due to which we are seeing issues. ---
**Author:** ![MarkEWaite](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/markewaite/32/20_2.png) [@MarkEWaite](https://community.jenkins.io/u/MarkEWaite)\ **Post date:** [May 6, 2024, 7:33pm UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/5 "2024-05-06T19:33:14Z")
> [@hpriya](#): > > `unable to create native thread` Some ideas that you might consider: - The out of memory exception is discussed in a [CloudBees knowledgebase article](https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-controllers/types-of-out-of-memory-exceptions) that might help - Java 11.0.15 was reported to have known issues with creating native threads. Java 11.0.16 has a known memory leak that is resolved in 11.0.16.1. Java 11.0.23 is the current Java 11 version released by OpenJDK. Upgrading the Java version may help. A [CloudBees knowledgebase article](https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/cloudbees-ci-on-modern-cloud-platforms/agent-sporadically-fails-with-unable-to-create-native-threads) provides more information and additional links - [JENKINS-65873](https://issues.jenkins.io/browse/JENKINS-65873) reports that thread exhaustion was an issue in older versions of [Jenkins remoting](https://github.com/jenkinsci/remoting/releases). More recent releases of Jenkins remoting may help. You could check to assure that your agents are running a recent release of Jenkins remoting. The [versions node monitors plugin](https://plugins.jenkins.io/versioncolumn/) can display the remoting version in the list of agents. - [Stackoverflow has an article](https://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread) that discusses different reasons what a Java process might be unable to create a new native thread ---
**Author:** ![hpriya](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hpriya](https://community.jenkins.io/u/hpriya)\ **Post date:** [May 7, 2024, 3:58am UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/6 "2024-05-07T03:58:58Z")
@MarkEWaite We have been following up with the stack-overflow thread and upgrading Java11.0.23. Thank you for the inputs. We will look further into it. ---
**Author:** ![hpriya](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hpriya](https://community.jenkins.io/u/hpriya)\ **Post date:** [May 7, 2024, 7:51am UTC](https://community.jenkins.io/t/can-jnlp-protocol-get-auto-disabled-due-to-jvm-memory-allocated/14831/7 "2024-05-07T07:51:41Z")
There is one more theory we ran into while investigating the issue regarding the ulimit set. We have the /etc/security/limit.conf set to ```auto sdaasbld soft nofile 8192 sdaasbld hard nofile 65536 sdaasbld soft nproc 8192 sdaasbld hard nproc 16384 ``` which points to > sdaasbld soft nofile 8192 ```auto [sdaasbld@ ~]$ ulimit -a max user processes (-u) 8192 ``` > sdaasbld soft nproc 8192 ```auto [sdaasbld@ ~]$ ulimit -u 8192 ``` We are seeing a spike in fileDescriptors when the event occurs which is \>16k ![filedescriptors-jenkins](https://europe1.discourse-cdn.com/flex013/uploads/jenkins/original/2X/1/16b1596878d077e8ef183b8a6b701654d71be2a5.png) ❓ Can this cause the misbehaviour of the `JNLP Connections` and `OutOfMemoryError` to create native threads?