Slowness of jenkins UI

Hi all

I have been facing slowness of jenkins UI only at certain times of a day . I was able to corelate that the jenkins got slow whenever the number of active threads spiked. I tried using several JVM option and tuned GC but nothing helped to resolve this . can someone help me to resolve this .

Jenkins setup:
Jenkins version - 2.440.2
java version 11
This is a large jenkins handling about 120+ spot agent nodes parallely. We have pipeline jobs and also also jobs that spin 80+ child jobs simultaneously.
we have our jenkins controller runnning in EKS with below JVM options:
-Xms10000m -Xmx20000m -XX:+UseG1GC -XX:+ParallelRefProcEnabled
-XX:+AlwaysPreTouch
-Xlog:gc*:file=/var/jenkins_home/gc.log:time,uptime,tags:filecount=5,filesize=10M
-Dhudson.slaves.NodeProvisioner.MARGIN=80
-Djenkins.slaves.Computer.threadPoolSize=750
-Dhudson.remoting.Engine.threadPoolSize=500
-Dwinstone.maxThreads=750
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85
-Dhudson.model.DirectoryBrowserSupport.CSP=“sandbox
allow-same-origin allow-scripts; default-src ‘self’;”