Jenkins huge CPU utilization provokes slowness

Hi Guys,

I have noticed some chokes = slowness on single node jenkins instance. I believe problem is related to inappropriate JVM params.
Instance is a bit of legacy running gui [Jenkins 2.303.2].
On prem server consists of 128GB RAM. CPU 64 . Instance has got 1000+ projects

openjdk version “1.8.0_161”
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

Quite recently “top” %CPU was reaching 6k during peak hours. I have swapped XMS quite recently to 32GB, because could see in the GC log "Full GC and Allocation Failure - heap already fully expanded once was XMS-16G XMX-32G)

I am also reviewing possible options

JENKINS_JAVA_OPTIONS="
-DSoftKillWaitSeconds=0
-Djava.awt.headless=true
-Djenkins.install.runSetupWizard=false
-DoctaneAllowedStorage=/apps/jenkins/
-Djava.io.tmpdir=/apps/jenkins/bin/temp
-Dhudson.model.DirectoryBrowserSupport.CSP=""
-Xms16g -Xmx32g -Xss1m -server
-XX:+UseG1GC
-XX:+ExplicitGCInvokesConcurrent
-XX:+ParallelRefProcEnabled
-XX:+UseStringDeduplication
-XX:+UnlockExperimentalVMOptions
-XX:G1NewSizePercent=40
-XX:+UnlockDiagnosticVMOptions
-XX:G1SummarizeRSetStatsPeriod=1
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=true
-Dcom.sun.management.jmxremote.password.file=/xx/jmxremote.password
-Dcom.sun.management.jmxremote.access.file=/xx/jmxremote.access
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=xxx
-Xloggc:/xx/gc-%t.log
-XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation
-XX:GCLogFileSize=20m
-XX:+PrintGC
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintHeapAtGC
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+PrintReferenceGC
-XX:+PrintAdaptiveSizePolicy
-Dhudson.upstreamCulprits=true
-Ddevops_pipeline_jenkins"

usually the recommended solution is to trigger a thread dump and take a look at the expert. Not many forum users have knowledge and patience for this advanced topic.

I know people have had luck with cloudbees support for this as they have tooling they can use to help analyze threaddumps, but its not something i’ve ever done perosnally.