Jenkins huge CPU utilization provokes slowness

Hi Guys,

I have noticed some chokes = slowness on single node jenkins instance. I believe problem is related to inappropriate JVM params.
Instance is a bit of legacy running gui [Jenkins 2.303.2].
On prem server consists of 128GB RAM. CPU 64 . Instance has got 1000+ projects

openjdk version “1.8.0_161”
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

Quite recently “top” %CPU was reaching 6k during peak hours. I have swapped XMS quite recently to 32GB, because could see in the GC log "Full GC and Allocation Failure - heap already fully expanded once was XMS-16G XMX-32G)

I am also reviewing possible options

-Xms16g -Xmx32g -Xss1m -server
-XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation

usually the recommended solution is to trigger a thread dump and take a look at the expert. Not many forum users have knowledge and patience for this advanced topic.

I know people have had luck with cloudbees support for this as they have tooling they can use to help analyze threaddumps, but its not something i’ve ever done perosnally.