Git plugin (git) and Number of active checkouts

Using Jenkins
1

Jenkins setup:
Jenkins Running on Openshift 3.11
Jenkins Updated to latest version
Git plugin (git) : 5.0.0

Agents used:
org.csanchez.jenkins.plugins.kubernetes.PodTemplate

We have a nightly build procedure were 50 jobs start building at the same time. The problem is that they don’t really start running more then 5-6 at a time.

The once that are not running har hanging in the initial SCM checkout on jenkins server. Before Agents are even started to be running.

Now when the initial 5-6 are passed the SCM checkout the agents start up and the next 5-6 jobs start to actually run the SCM checkout.

We have tried with:

SSH has a default of 10 connections

SCM Polling
Max # of concurrent polling set to 30

Anyone have any idea of how to increase the number of parallel Git Plugin initial checkouts ?

2

Hello @henriksp and welcome to this community. :wave:

Here are a few steps you could maybe take to troubleshoot this issue:

  1. Check the Git server configuration: The Git server might have a limit on the number of concurrent connections it allows. If this limit is set to a low value, it can cause delays in Git operations when multiple jobs are running concurrently. You might need to increase this limit on the Git server.
  2. Check the Jenkins system configuration: Jenkins has a system configuration option named Max # of concurrent builds that controls the maximum number of concurrent builds that Jenkins will perform. If this value is set to a low number, it can limit the number of jobs that can run concurrently. You might need to increase this value in the Jenkins system configuration.
  3. Check the system resources on the Jenkins server: If the Jenkins server is running out of resources (CPU, memory, disk I/O, network bandwidth, etc.), it can slow down the execution of jobs and cause delays in Git operations. You might need to increase the resources available to the Jenkins server.
  4. Check the Jenkins logs: The Jenkins logs can provide useful information about what’s happening when the jobs are running. Look for any error messages or warnings that might indicate what’s causing the delay in the Git checkouts.
3

Hi

  1. when we see the issue we don’t have problems with download from Git server else where
  2. I believe this is set to 0 but even though it’s 0 it still execute the checkouts
  3. the VM has enough resources and the Jenkins instanse is not hitting the max for resources. However we sometimes see the Vm Host is loaded.
  4. When it happens sometimes we see errors like this:

SEVERE io.jenkins.blueocean.autofavorite.FavoritingScmListener onCheckout Unexpected error when retrieving changeset
hudson.plugins.git.GitException: Error: git whatchanged --no-abbrev -M “–format=commit %H%ntree %T%nparent %P%nauthor %aN <%aE> %ai%ncommitter %cN <%cE> %ci%n%n%w(0,4,4)%B” -n 1

I have tried to increase bitbucket timeout from 10 to 30. But multibranch pipelines ignores it. I have also tried with JVM arg:
-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=30

Update:

I managed to configure the jenkins pod and making it used a other timeout then 10 min.

Here is my deployment.yaml config for this timeout setup:

containers:
- name: jenkins
image: jenkins/jenkins:lts
env:
- name: JAVA_OPTS
value: >-
-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=33
resources:
limits:
memory: “8Gi”
cpu: “5”
requests:
memory: “2000Mi”
cpu: “2”