Jenkins Fails to pull from github randomly

Jenkins setup:
Running in docker container behind nginx reverse proxy.
Version 2.426.3

Hi everyone I randomly get this error when trying to pull from Github, seems to happen across multiple different workers and deleting the tmp folder in the workspace seems to fix it temporarily. We do have a cached github repo locally on each runner that is updated periodically, but that has been in place for a few months without issues.

g objects:  54% (51/93)        
remote: Compressing objects:  55% (52/93)        
remote: Compressing objects:  56% (53/93)        
remote: Compressing objects:  58% (54/93)        
remote: Compressing objects:  59% (55/93)        
remote: Compressing objects:  60% (56/93)        
remote: Compressing objects:  61% (57/93)        
remote: Compressing objects:  62% (58/93)        
remote: Compressing objects:  63% (59/93)        
remote: Compressing objects:  64% (60/93)        
remote: Compressing objects:  65% (61/93)        
remote: Compressing objects:  66% (62/93)        
remote: Compressing objects:  67% (63/93)        
remote: Compressing objects:  68% (64/93)        
remote: Compressing objects:  69% (65/93)        
remote: Compressing objects:  70% (66/93)        
remote: Compressing objects:  72% (67/93)        
remote: Compressing objects:  73% (68/93)        
remote: Compressing objects:  74% (69/93)        
remote: Compressing objects:  75% (70/93)        
remote: Compressing objects:  76% (71/93)        
remote: Compressing objects:  77% (72/93)        
remote: Compressing objects:  78% (73/93)        
remote: Compressing objects:  79% (74/93)        
remote: Compressing objects:  80% (75/93)        
remote: Compressing objects:  81% (76/93)        
remote: Compressing objects:  82% (77/93)        
remote: Compressing objects:  83% (78/93)        
remote: Compressing objects:  84% (79/93)        
remote: Compressing objects:  86% (80/93)        
remote: Compressing objects:  87% (81/93)        
remote: Compressing objects:  88% (82/93)        
remote: Compressing objects:  89% (83/93)        
remote: Compressing objects:  90% (84/93)        
remote: Compressing objects:  91% (85/93)        
remote: Compressing objects:  92% (86/93)        
remote: Compressing objects:  93% (87/93)        
remote: Compressing objects:  94% (88/93)        
remote: Compressing objects:  95% (89/93)        
remote: Compressing objects:  96% (90/93)        
remote: Compressing objects:  97% (91/93)        
remote: Compressing objects:  98% (92/93)        
remote: Compressing objects: 100% (93/93)        
remote: Compressing objects: 100% (93/93), done.        
Receiving objects:   0% (1/127)
Receiving objects:   1% (2/127)
Receiving objects:   2% (3/127)
Receiving objects:   3% (4/127)
Receiving objects:   4% (6/127)
Receiving objects:   5% (7/127)
Receiving objects:   6% (8/127)
Receiving objects:   7% (9/127)
Receiving objects:   8% (11/127)
Receiving objects:   9% (12/127)
Receiving objects:  10% (13/127)
Receiving objects:  11% (14/127)
Receiving objects:  12% (16/127)
Receiving objects:  13% (17/127)
Receiving objects:  14% (18/127)
Receiving objects:  15% (20/127)
Receiving objects:  16% (21/127)
Receiving objects:  17% (22/127)
Receiving objects:  18% (23/127)
Receiving objects:  19% (25/127)
Receiving objects:  20% (26/127)
Receiving objects:  21% (27/127)
Receiving objects:  22% (28/127)
Receiving objects:  23% (30/127)
Receiving objects:  24% (31/127)
Receiving objects:  25% (32/127)
Receiving objects:  26% (34/127)
Receiving objects:  27% (35/127)
Receiving objects:  28% (36/127)
Receiving objects:  29% (37/127)
Receiving objects:  30% (39/127)
Receiving objects:  31% (40/127)
Receiving objects:  32% (41/127)
Receiving objects:  33% (42/127)
Receiving objects:  34% (44/127)
Receiving objects:  35% (45/127)
Receiving objects:  36% (46/127)
Receiving objects:  37% (47/127)
Receiving objects:  38% (49/127)
Receiving objects:  39% (50/127)
Receiving objects:  40% (51/127)
Receiving objects:  41% (53/127)
Receiving objects:  42% (54/127)
Receiving objects:  43% (55/127)
Receiving objects:  44% (56/127)
Receiving objects:  45% (58/127)
Receiving objects:  46% (59/127)
Receiving objects:  47% (60/127)
Receiving objects:  48% (61/127)
Receiving objects:  49% (63/127)
Receiving objects:  50% (64/127)
Receiving objects:  51% (65/127)
Receiving objects:  52% (67/127)
Receiving objects:  53% (68/127)
Receiving objects:  54% (69/127)
Receiving objects:  55% (70/127)
Receiving objects:  56% (72/127)
Receiving objects:  57% (73/127)
Receiving objects:  58% (74/127)
Receiving objects:  59% (75/127)
Receiving objects:  60% (77/127)
Receiving objects:  61% (78/127)
Receiving objects:  62% (79/127)
Receiving objects:  63% (81/127)
Receiving objects:  64% (82/127)
Receiving objects:  65% (83/127)
Receiving objects:  66% (84/127)
Receiving objects:  67% (86/127)
Receiving objects:  68% (87/127)
Receiving objects:  69% (88/127)
Receiving objects:  70% (89/127)
Receiving objects:  71% (91/127)
Receiving objects:  72% (92/127)
Receiving objects:  73% (93/127)
Receiving objects:  74% (94/127)
Receiving objects:  75% (96/127)
Receiving objects:  76% (97/127)
Receiving objects:  77% (98/127)
Receiving objects:  78% (100/127)
Receiving objects:  79% (101/127)
Receiving objects:  80% (102/127)
Receiving objects:  81% (103/127)
Receiving objects:  82% (105/127)
remote: Total 127 (delta 45), reused 94 (delta 26), pack-reused 0        
Receiving objects:  83% (106/127)
Receiving objects:  84% (107/127)
Receiving objects:  85% (108/127)
Receiving objects:  86% (110/127)
Receiving objects:  87% (111/127)
Receiving objects:  88% (112/127)
Receiving objects:  89% (114/127)
Receiving objects:  90% (115/127)
Receiving objects:  91% (116/127)
Receiving objects:  92% (117/127)
Receiving objects:  93% (119/127)
Receiving objects:  94% (120/127)
Receiving objects:  95% (121/127)
Receiving objects:  96% (122/127)
Receiving objects:  97% (124/127)
Receiving objects:  98% (125/127)
Receiving objects:  99% (126/127)
Receiving objects: 100% (127/127)
Receiving objects: 100% (127/127), 143.67 KiB | 510.00 KiB/s, done.
Resolving deltas:   0% (0/45)
Resolving deltas:   2% (1/45)
Resolving deltas:   4% (2/45)
Resolving deltas:   6% (3/45)
Resolving deltas:   8% (4/45)
Resolving deltas:  11% (5/45)
Resolving deltas:  13% (6/45)
Resolving deltas:  15% (7/45)
Resolving deltas:  17% (8/45)
Resolving deltas:  20% (9/45)
Resolving deltas:  22% (10/45)
Resolving deltas:  24% (11/45)
Resolving deltas:  26% (12/45)
Resolving deltas:  28% (13/45)
Resolving deltas:  31% (14/45)
Resolving deltas:  33% (15/45)
Resolving deltas:  35% (16/45)
Resolving deltas:  37% (17/45)
Resolving deltas:  40% (18/45)
Resolving deltas:  42% (19/45)
Resolving deltas:  44% (20/45)
Resolving deltas:  46% (21/45)
Resolving deltas:  48% (22/45)
Resolving deltas:  51% (23/45)
Resolving deltas:  53% (24/45)
Resolving deltas:  55% (25/45)
Resolving deltas:  57% (26/45)
Resolving deltas:  60% (27/45)
Resolving deltas:  62% (28/45)
Resolving deltas:  64% (29/45)
Resolving deltas:  66% (30/45)
Resolving deltas:  68% (31/45)
Resolving deltas:  71% (32/45)
Resolving deltas:  73% (33/45)
Resolving deltas:  75% (34/45)
Resolving deltas:  77% (35/45)
Resolving deltas:  80% (36/45)
Resolving deltas:  82% (37/45)
Resolving deltas:  84% (38/45)
Resolving deltas:  86% (39/45)
Resolving deltas:  88% (40/45)
Resolving deltas:  91% (41/45)
Resolving deltas:  93% (42/45)
Resolving deltas:  95% (43/45)
Resolving deltas:  97% (44/45)
Resolving deltas: 100% (45/45)
Resolving deltas: 100% (45/45), completed with 18 local objects.
fatal: bad object refs/remotes/origin/my-branch
error: github.com:myrepo.git did not send all necessary objects


	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2842)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:635)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Jenkins Worker 03
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1003)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at jdk.internal.reflect.GeneratedMethodAccessor339.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:568)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at jdk.proxy36/jdk.proxy36.$Proxy113.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:997)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1241)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)

If deleting the tmp folder fixes the issue temporarily, have you looked into how much disk space your agents have and whether they’re running out?

I had a permanent agent with a weird version of Git that left my .git leaking disk space. I found that by starting to investigate where all my disk space was going.