How to optimise Jenkins Remoting for SCM plugin

Jenkins setup: 2.303.3-jdk8

My setup involves running controller from my local machine and using kubernetes plugin to spawn agents in kubernetes cluster in production

I have got this setup using ktunnel

I read through how remoting works and hence cached the remoting jars in a persistent volume in my kubernetes cluster, and expecting them to load once by agent and next time the build should have been faster. But the SCM step takes a lot of time, for a simple step it takes 3 minutes to move to the next step.

The ping latency between my local machine and the cluster is 250 ms. Is there a way I can optimise this?

Note that I have even tried the setup with latest jenkins version, but it doesn’t still resolve the issue