Jenkins Fails to pull from github randomly

Using Jenkins
1

Jenkins setup:
Running in docker container behind nginx reverse proxy.
Version 2.426.3

Hi everyone I randomly get this error when trying to pull from Github, seems to happen across multiple different workers and deleting the tmp folder in the workspace seems to fix it temporarily. We do have a cached github repo locally on each runner that is updated periodically, but that has been in place for a few months without issues.

g objects:  54% (51/93)        
remote: Compressing objects:  55% (52/93)        
remote: Compressing objects:  56% (53/93)        
remote: Compressing objects:  58% (54/93)        
remote: Compressing objects:  59% (55/93)        
remote: Compressing objects:  60% (56/93)        
remote: Compressing objects:  61% (57/93)        
remote: Compressing objects:  62% (58/93)        
remote: Compressing objects:  63% (59/93)        
remote: Compressing objects:  64% (60/93)        
remote: Compressing objects:  65% (61/93)        
remote: Compressing objects:  66% (62/93)        
remote: Compressing objects:  67% (63/93)        
remote: Compressing objects:  68% (64/93)        
remote: Compressing objects:  69% (65/93)        
remote: Compressing objects:  70% (66/93)        
remote: Compressing objects:  72% (67/93)        
remote: Compressing objects:  73% (68/93)        
remote: Compressing objects:  74% (69/93)        
remote: Compressing objects:  75% (70/93)        
remote: Compressing objects:  76% (71/93)        
remote: Compressing objects:  77% (72/93)        
remote: Compressing objects:  78% (73/93)        
remote: Compressing objects:  79% (74/93)        
remote: Compressing objects:  80% (75/93)        
remote: Compressing objects:  81% (76/93)        
remote: Compressing objects:  82% (77/93)        
remote: Compressing objects:  83% (78/93)        
remote: Compressing objects:  84% (79/93)        
remote: Compressing objects:  86% (80/93)        
remote: Compressing objects:  87% (81/93)        
remote: Compressing objects:  88% (82/93)        
remote: Compressing objects:  89% (83/93)        
remote: Compressing objects:  90% (84/93)        
remote: Compressing objects:  91% (85/93)        
remote: Compressing objects:  92% (86/93)        
remote: Compressing objects:  93% (87/93)        
remote: Compressing objects:  94% (88/93)        
remote: Compressing objects:  95% (89/93)        
remote: Compressing objects:  96% (90/93)        
remote: Compressing objects:  97% (91/93)        
remote: Compressing objects:  98% (92/93)        
remote: Compressing objects: 100% (93/93)        
remote: Compressing objects: 100% (93/93), done.        
Receiving objects:   0% (1/127)
Receiving objects:   1% (2/127)
Receiving objects:   2% (3/127)
Receiving objects:   3% (4/127)
Receiving objects:   4% (6/127)
Receiving objects:   5% (7/127)
Receiving objects:   6% (8/127)
Receiving objects:   7% (9/127)
Receiving objects:   8% (11/127)
Receiving objects:   9% (12/127)
Receiving objects:  10% (13/127)
Receiving objects:  11% (14/127)
Receiving objects:  12% (16/127)
Receiving objects:  13% (17/127)
Receiving objects:  14% (18/127)
Receiving objects:  15% (20/127)
Receiving objects:  16% (21/127)
Receiving objects:  17% (22/127)
Receiving objects:  18% (23/127)
Receiving objects:  19% (25/127)
Receiving objects:  20% (26/127)
Receiving objects:  21% (27/127)
Receiving objects:  22% (28/127)
Receiving objects:  23% (30/127)
Receiving objects:  24% (31/127)
Receiving objects:  25% (32/127)
Receiving objects:  26% (34/127)
Receiving objects:  27% (35/127)
Receiving objects:  28% (36/127)
Receiving objects:  29% (37/127)
Receiving objects:  30% (39/127)
Receiving objects:  31% (40/127)
Receiving objects:  32% (41/127)
Receiving objects:  33% (42/127)
Receiving objects:  34% (44/127)
Receiving objects:  35% (45/127)
Receiving objects:  36% (46/127)
Receiving objects:  37% (47/127)
Receiving objects:  38% (49/127)
Receiving objects:  39% (50/127)
Receiving objects:  40% (51/127)
Receiving objects:  41% (53/127)
Receiving objects:  42% (54/127)
Receiving objects:  43% (55/127)
Receiving objects:  44% (56/127)
Receiving objects:  45% (58/127)
Receiving objects:  46% (59/127)
Receiving objects:  47% (60/127)
Receiving objects:  48% (61/127)
Receiving objects:  49% (63/127)
Receiving objects:  50% (64/127)
Receiving objects:  51% (65/127)
Receiving objects:  52% (67/127)
Receiving objects:  53% (68/127)
Receiving objects:  54% (69/127)
Receiving objects:  55% (70/127)
Receiving objects:  56% (72/127)
Receiving objects:  57% (73/127)
Receiving objects:  58% (74/127)
Receiving objects:  59% (75/127)
Receiving objects:  60% (77/127)
Receiving objects:  61% (78/127)
Receiving objects:  62% (79/127)
Receiving objects:  63% (81/127)
Receiving objects:  64% (82/127)
Receiving objects:  65% (83/127)
Receiving objects:  66% (84/127)
Receiving objects:  67% (86/127)
Receiving objects:  68% (87/127)
Receiving objects:  69% (88/127)
Receiving objects:  70% (89/127)
Receiving objects:  71% (91/127)
Receiving objects:  72% (92/127)
Receiving objects:  73% (93/127)
Receiving objects:  74% (94/127)
Receiving objects:  75% (96/127)
Receiving objects:  76% (97/127)
Receiving objects:  77% (98/127)
Receiving objects:  78% (100/127)
Receiving objects:  79% (101/127)
Receiving objects:  80% (102/127)
Receiving objects:  81% (103/127)
Receiving objects:  82% (105/127)
remote: Total 127 (delta 45), reused 94 (delta 26), pack-reused 0        
Receiving objects:  83% (106/127)
Receiving objects:  84% (107/127)
Receiving objects:  85% (108/127)
Receiving objects:  86% (110/127)
Receiving objects:  87% (111/127)
Receiving objects:  88% (112/127)
Receiving objects:  89% (114/127)
Receiving objects:  90% (115/127)
Receiving objects:  91% (116/127)
Receiving objects:  92% (117/127)
Receiving objects:  93% (119/127)
Receiving objects:  94% (120/127)
Receiving objects:  95% (121/127)
Receiving objects:  96% (122/127)
Receiving objects:  97% (124/127)
Receiving objects:  98% (125/127)
Receiving objects:  99% (126/127)
Receiving objects: 100% (127/127)
Receiving objects: 100% (127/127), 143.67 KiB | 510.00 KiB/s, done.
Resolving deltas:   0% (0/45)
Resolving deltas:   2% (1/45)
Resolving deltas:   4% (2/45)
Resolving deltas:   6% (3/45)
Resolving deltas:   8% (4/45)
Resolving deltas:  11% (5/45)
Resolving deltas:  13% (6/45)
Resolving deltas:  15% (7/45)
Resolving deltas:  17% (8/45)
Resolving deltas:  20% (9/45)
Resolving deltas:  22% (10/45)
Resolving deltas:  24% (11/45)
Resolving deltas:  26% (12/45)
Resolving deltas:  28% (13/45)
Resolving deltas:  31% (14/45)
Resolving deltas:  33% (15/45)
Resolving deltas:  35% (16/45)
Resolving deltas:  37% (17/45)
Resolving deltas:  40% (18/45)
Resolving deltas:  42% (19/45)
Resolving deltas:  44% (20/45)
Resolving deltas:  46% (21/45)
Resolving deltas:  48% (22/45)
Resolving deltas:  51% (23/45)
Resolving deltas:  53% (24/45)
Resolving deltas:  55% (25/45)
Resolving deltas:  57% (26/45)
Resolving deltas:  60% (27/45)
Resolving deltas:  62% (28/45)
Resolving deltas:  64% (29/45)
Resolving deltas:  66% (30/45)
Resolving deltas:  68% (31/45)
Resolving deltas:  71% (32/45)
Resolving deltas:  73% (33/45)
Resolving deltas:  75% (34/45)
Resolving deltas:  77% (35/45)
Resolving deltas:  80% (36/45)
Resolving deltas:  82% (37/45)
Resolving deltas:  84% (38/45)
Resolving deltas:  86% (39/45)
Resolving deltas:  88% (40/45)
Resolving deltas:  91% (41/45)
Resolving deltas:  93% (42/45)
Resolving deltas:  95% (43/45)
Resolving deltas:  97% (44/45)
Resolving deltas: 100% (45/45)
Resolving deltas: 100% (45/45), completed with 18 local objects.
fatal: bad object refs/remotes/origin/my-branch
error: github.com:myrepo.git did not send all necessary objects


	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Jenkins Worker 03
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1003)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at jdk.internal.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:568)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at jdk.proxy36/jdk.proxy36.$Proxy113.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1241)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)
1 Like
2

If deleting the tmp folder fixes the issue temporarily, have you looked into how much disk space your agents have and whether they’re running out?

I had a permanent agent with a weird version of Git that left my .git leaking disk space. I found that by starting to investigate where all my disk space was going.

3

Same happens (suddenly) on all of my pipelines (connected to GitHub).

  • GitVersion5.5.2
  • Jenkins 2.462.2
  • We are using a Git mirror which is updated daily

If I execute natively the git command on the VM

git fetch --tags --force --progress --prune -- https://github.com/GIT_ORG/REPO_NAME.git +refs/heads/*:refs/remotes/origin/* # timeout=10

the same error (fatal: bad object refs/remotes/origin) occurs, so this is not a Git plugin problem itself.

It would be great, if the plugin could provide a solution to such “hickups”.

The only workaround for me is to delete the .git dir and to restart the pipeline.

This is my JobDsl config

scm {
  git {
    remote {
      github('GIT_ORG/REPO_NAME')
      credentials('github-id')
    }
    branch('${BRANCH}')
    extensions {
      cloneOptions {
        cleanBeforeCheckout()
        reference('/gitcache/GIT_ORG/REPO_NAME.git')
        honorRefspec()
      }
    }
  }
}

Unfortunately, a full workspace wipe-out is not possible in my case (which would solve the problem as well).

I did not found any Jira issue

The log states

The recommended git tool is: NONE
using credential github
Fetching changes from the remote Git repository
Pruning obsolete local branches
Pruning obsolete local tags that do not exist on remotes
 > git rev-parse --resolve-git-dir /home/test-user/my-pipeline/.git # timeout=10
 > git config remote.origin.url https://github.com/GIT_ORG/REPO_NAME.git # timeout=10
 > git show-ref --tags -d # timeout=10
Fetching upstream changes from https://github.com/GIT_ORG/REPO_NAME.git
 > git --version # timeout=10
 > git --version # 'git version 2.42.0'
using GIT_ASKPASS to set credentials github personal access token (ci-stack) for jenkins setup via https://github.com/GIT_ORG/OTHER_REPO/tree/shared-lib

formerly: cm-jenkins
 > git fetch --tags --force --progress --prune -- https://github.com/GIT_ORG/REPO_NAME.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/GIT_ORG/REPO_NAME.git
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
	at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
	at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress --prune -- https://github.com/GIT_ORG/REPO_NAME.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
stdout: 
stderr: remote: Enumerating objects: 1909, done.        
remote: Counting objects:   0% (1/1903)        
[...]   
remote: Counting objects: 100% (1903/1903), done.        
remote: Compressing objects:   0% (1/598)        
remote: Compressing objects:   1% (6/598)        
[...]  
remote: Compressing objects: 100% (598/598)        
remote: Compressing objects: 100% (598/598), done.        
Receiving objects:   0% (1/1909)
[...]
Receiving objects:  92% (1757/1909)
remote: Total 1909 (delta 1133), reused 1811 (delta 1055), pack-reused 6 (from 1)        
Receiving objects:  93% (1776/1909)
[...]
Receiving objects: 100% (1909/1909)
Receiving objects: 100% (1909/1909), 344.04 KiB | 22.94 MiB/s, done.
Resolving deltas:   0% (0/1133)
Resolving deltas:   1% (12/1133)
[...]
Resolving deltas:  99% (1122/1133)
Resolving deltas: 100% (1133/1133)
Resolving deltas: 100% (1133/1133), completed with 80 local objects.
fatal: bad object refs/remotes/origin/CORRUPTED_BRANCH
error: https://github.com/GIT_ORG/REPO_NAME.git did not send all necessary objects


	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to MY_BUILD_NODE
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1826)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1042)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at jdk.internal.reflect.GeneratedMethodAccessor963.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
		at java.base/java.lang.reflect.Method.invoke(Unknown Source)
		at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at PluginClassLoader for git-client/jdk.proxy75/jdk.proxy75.$Proxy171.execute(Unknown Source)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:995)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1239)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM._checkout(GitSCM.java:1310)
		at PluginClassLoader for git//hudson.plugins.git.GitSCM.checkout(GitSCM.java:1277)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:136)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:101)
		at PluginClassLoader for workflow-scm-step//org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:88)
		at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
ERROR: Error fetching remote repo 'origin'