Intermittent failures connecting to GitHub

Using Jenkins Ask a question
1

Our Jenkins (stand-alone) instance is located behind a firewall. We connect to a variety Git repository hosts, including on-prem TFS and GitHub. Our GitHub repos are internal/private to the Enterprise.

On occasion, certain freestyle jobs which have run fine in the past suddenly fail very quickly on the Git checkout step with `

stderr: remote: Repository not found

Rerunning the job (dispatched to the same node) with no changes usually results in subsequent success. Unfortunately the logs do not shed much light on what the error may be.

Admittedly, the occurrence is low, approximately 1% of the total, but 0.25% if you treat the repeated failures as a single interval. But when it comes up, it’s annoying!

Any suggestion on how/where to enable more detailed logging to trace the root of the problem?

[EnvInject] - Loading node environment variables.
Building remotely on LNX_A (linux) in workspace /work/MyJob/BAU
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/myorg/myrepo.git # timeout=10
Fetching upstream changes from https://github.com/myorg/myrepo.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials GitHub service account using token created with myorggithubbot account
Setting http proxy: pac.myorg.com:8080
 > git fetch --tags --progress https://github.com/myorg/myrepo.git +refs/heads/*:refs/remotes/origin/* --depth=1 # timeout=45
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/myorg/myrepo.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
	at hudson.scm.SCM.checkout(SCM.java:504)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
	at hudson.model.Run.execute(Run.java:1794)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/myorg/myrepo.git +refs/heads/*:refs/remotes/origin/* --depth=1" returned status code 128:
stdout: 
stderr: remote: Repository not found.
fatal: repository 'https://github.com/myorg/myrepo.git/' not found

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2002)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1721)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:405)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to LNX_A
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:955)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
		at sun.reflect.GeneratedMethodAccessor377.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
		at com.sun.proxy.$Proxy86.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:886)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
		at hudson.scm.SCM.checkout(SCM.java:504)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
		at hudson.model.Run.execute(Run.java:1794)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
2

Since the GitHub server is inside your firewall, I think that you’ll need to review logs from the GitHub server at the time of the failure. The remote message means that the remote server returned that message. There is probably a log file written somewhere that explains why the repository was temporarily not found.

3

Thanks for the response, @MarkEWaite.

To clarify though, Jenkins and TFS are on-prem behind the firewall, but GitHub is github.com, aka Enterprise Cloud, with Enterprise license, and our repos set to Internal / Private (ie: not Public). (GitHub also calls their on-prem variant Enterprise Server).

Connectivity works 99+% of the time, so it’s not a typical “configuration problem.” A previously working job will just fail for a bit then work again, unchanged.

The question whether there is some debugging we can enable to verbose logging of the git plugin within Jenkins or pass the GIT_TRACE flags to git so we can get some insight into what’s actually happening, what’s causing the status code 128

4

Since the message is coming from the remote side (github.com), I think that additional tracing on the agent side (inside Jenkins) is unlikely to provide the additional details that you hope. However, there is an article that describes how to enable git tracing for a Jenkins job.

1 Like