Timeout connecting to GitHub

I am also getting this issue after upgrading, but is of a slightly different error. I am also running this on a Windows secondary node.

Here’s some information on the environment:

  • Jenkins 2.332.1
  • Credentials Plugin Version 1087.v16065d268466
  • SSH Credentials Plugin Version 1.19
  • Git plugin Version 4.11.0
  • Git client plugin Version 3.11.0
  • GIT server Plugin Version 1.10

Log:

Building remotely on windows_server_2019_main (windows) in workspace C:\JenkinsWorkspace\Test/GitHub-Test-17
The recommended git tool is: NONE
using credential devel-ed25519
Cloning the remote Git repository
Cloning repository <repository>.git
 > C:\Program Files\Git\bin\git.exe init C:\JenkinsWorkspace\Test\GitHub-Test-17 # timeout=10
Fetching upstream changes from <repository>.git
 > C:\Program Files\Git\bin\git.exe --version # timeout=10
 > git --version # 'git version 2.27.0.windows.1'
using GIT_SSH to set credentials devel ed25519
 > C:\Program Files\Git\bin\git.exe fetch --tags --force --progress -- <repository>.git +refs/heads/*:refs/remotes/origin/* # timeout=10

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "C:\Program Files\Git\bin\git.exe fetch --tags --force --progress -- <repository>.git +refs/heads/*:refs/remotes/origin/*" returned status code 1:
stdout: 
stderr: remote: Enumerating objects: 75124, done.     

...


	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:847)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:376)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
	at java.lang.Thread.run(Unknown Source)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from <server>.com/10.51.13.115:55598
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1785)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1000)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
		at com.sun.proxy.$Proxy80.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1216)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1294)
		at hudson.scm.SCM.checkout(SCM.java:540)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1215)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
		at hudson.model.Run.execute(Run.java:1896)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
		at hudson.model.ResourceController.execute(ResourceController.java:101)
		at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE

so to add to the trace git, i seem to have discovered in the Source Code Management section there’s an error for Failed to connect to repository : Command "git ls-remote -h -- git@<repository>.git HEAD" returned status code 128

i looked at the log, it seem like it’s expecting an ssh key from the default execution user of jenkins? i got an error for looking for ssh key in a separate directory and not credential i set?

debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ed25519-cert type -1 

The issue you are reporting is a completely different issue. @DappleZapple reports that authentication fails in a git ls-remote call. You are reporting that after authenticating and after listing some objects from the remote repository, something fails. It could be a timeout. It could be a problem on the remote server. It is not an authentication issue. The problem you’re seeing is after the authentication has completed and data transfer has happened between the remote and the local git client.

This is a different topic than the issue that @DappleZapple is investigating.

Per marks comments I split this into a new topic

I believe they might also be similar issues, because I am also getting failed git ls-remote call, which lead to the Error cloning remote repo issue. So let me provide some clarification.

The error log from the first post above is from when I executed the job itself.

After following the How to trace git connections article you’ve provided, when I look at the Source Code Management section of the job, it gave me the authentication issue with git ls-remote. I’ll post the full log below:

Failed to connect to repository : Command "git ls-remote -h -- git@<repository>.git HEAD" returned status code 128:
stdout:
stderr: 17:14:36.794132 git.c:444               trace: built-in: git ls-remote -h -- git@<repository>.git HEAD
17:14:36.794357 run-command.c:663       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 'ssh -vvv' -o SendEnv=GIT_PROTOCOL git@<server> 'git-upload-pack '\''<repository>.git'\'''
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug2: resolving "<server>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <server> [10.50.226.16] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/<user>/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version babeld-eb8eb2c
debug1: no match: babeld-eb8eb2c
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to <server>:22 as 'git'
debug3: hostkeys_foreach: reading file "/apps/home/<user>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/<user>/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <server>
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1-etm@openssh.com,hmac-sha1
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1-etm@openssh.com,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:7J1hDzoAs0vF9KOR1KbzMijEBIF4f0P2cYVzIU9zDJs
debug3: hostkeys_foreach: reading file "/apps/home/<user>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/<user>/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <server>
debug3: hostkeys_foreach: reading file "/apps/home/<user>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/<user>/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 10.50.226.16
debug1: Host '<server>' is known and matches the ECDSA host key.
debug1: Found key in /apps/home/<user>/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /apps/home/<user>/.ssh/id_rsa ((nil))
debug2: key: /apps/home/<user>/.ssh/id_dsa ((nil))
debug2: key: /apps/home/<user>/.ssh/id_ecdsa ((nil))
debug2: key: /apps/home/<user>/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /apps/home/<user>/.ssh/id_rsa
debug3: no such identity: /apps/home/<user>/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /apps/home/<user>/.ssh/id_dsa
debug3: no such identity: /apps/home/<user>/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /apps/home/<user>/.ssh/id_ecdsa
debug3: no such identity: /apps/home/<user>/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /apps/home/<user>/.ssh/id_ed25519
debug3: no such identity: /apps/home/<user>/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
17:14:36.872116 trace.c:487             performance: 0.078082367 s: git command: git ls-remote -h -- git@<repository>.git HEAD

I’ve replied with a post, but seem to be filtered, can someone review the post?

https://community.jenkins.io/t/timeout-connecting-to-github/2007/5

I’m not understanding what you are saying. You seem to be making statements that seem contradictory to me.

In your first message, you said that the error was:

The message with “Enumerating objects” means that communication was established from the git process on the Jenkins agent to the git server. That communication was enough that it allowed the git process on the Jenkins agent to list 75124 objects on the git server. That means the connection was authenticated successfully and the git process on the Jenkins agent received the list of objects from the git server.

Then you said in a later message:

That indicates the call to ls-remote failed. If the ls-remote failed, then I would expect the fetch to fail as well. Yet, your first message indicates that the fetch received data from the git server.

Please clarify the contradiction between the original message and the most recent message.

Are you using Jenkins credentials? If not, please do that. Relying on the ssh keys stored on the controller file system and the agent file system is an invitation for surprises and problems.

You indicate that you are surprised that the ls-remote allows use of the ssh private keys of the controller user. That’s an allowed usage, though not a recommended usage. Some users have chosen to rely on the ssh private keys on the controller file system rather than using the Jenkins credentials system.

Apologies Mark, I understand there is a contradiction, I am also a bit confused as well. So let me try to clarify some points you’ve mentioned.

I am currently using Jenkins credentials, it is set under Global credentials as SSH Username with private key.

From here I created just a simple job just to pull from Git under the Source Code Management section of the job. I added the Repository URL and selected the Credentials, which is the SSH key I’ve created above. Usually when you select the Credentials, it provide a feedback whether the key is valid. Before Jenkins 2.332.1 I never had issues. But I got the git ls-remote error mentioned above after upgrade to Jenkins 2.332.1. I’ve verified the SSH key works on both my local machine as well as the Jenkins server I am running the job on. I’ve also try 2 different keys, one for rsa and one for ed25519, both same result.

If I ignore the error above and just run the job. It gave the first error I had with ERROR: Error cloning remote repo 'origin'.

It is possible that because of how our environment is setup, Jenkins is doing the git ls-remote and C:\Program Files\Git\bin\git.exe fetch command differently. It might be because our primary node is running RedHat, and the secondary node where the C:\Program Files\Git\bin\git.exe fetch command resides is Windows. This might be where the contradiction is?

But I tried just running the same command C:\Program Files\Git\bin\git.exe fetch --tags --force --progress -- <repository>.git +refs/heads/*:refs/remotes/origin/* # timeout=10 manually on the Windows node it seemed to work fine, but only through Jenkins that I am getting this issue.

I can’t duplicate the problem. Here are the steps that I tried:

  1. Define a passphrase protected ed25519 ssh private key on a fully patched Red Hat Enterprise Linux 8 system
  2. Register the public key for that ed25519 ssh private key with my GitHub account
  3. Create a Jenkins freestyle job on a Red Hat Linux 8 system running Jenkins 2.341 to clone a private GitHub repository git@github.com:MarkEWaite/bin.git
  4. Confirm that without a private key, the job creation page correctly reports permission denied
  5. Define a Jenkins credential with the passphrase protected ed25519 ssh private key
  6. Use the Jenkins credential in the freestyle job definition to confirm that the permission is no longer denied
  7. Run the freestyle job to confirm that the repository can be cloned
  8. Configure the freestyle job to only run on agents with the label windows
  9. Run the freestyle job to confirm that the repository can be cloned on a Windows agent

Since you said that you’re seeing the failure with an ed25519 key and with an rsa key, I took the easy way out and used an ed25519. You might review the improving Git protocol security blog post in case any of those changes might affect your version of Red Hat Linux.

Initially the job was restricted to build to the Windows node, but I switch it over to the primary RedHat node, it just gave me the code 128 error.

I’ll also consult with out infra team to see if is anything on the infra side.

Building on the built-in node in workspace /apps/jenkins/latest/workspace/Test/GitHub-Test
The recommended git tool is: NONE
using credential devel-ed25519
Cloning the remote Git repository
Cloning repository git@<repository>.git
 > git init /apps/jenkins/latest/workspace/Test/GitHub-Test # timeout=10
Fetching upstream changes from git@<repository>.git
 > git --version # timeout=10
 > git --version # 'git version 2.29.2'
using GIT_SSH to set credentials devel ed25519
[INFO] Currently running in a labeled security context
[INFO] Currently SELinux is 'enforcing' on the host
 > /usr/bin/chcon --type=ssh_home_t /apps/jenkins/latest/workspace/Test/GitHub-Test@tmp/jenkins-gitclient-ssh3158282332715523936.key
 > git fetch --tags --force --progress -- git@<repository>.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@<repository>.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: 03:59:01.079849 git.c:444               trace: built-in: git fetch --tags --force --progress -- git@<repository>.git '+refs/heads/*:refs/remotes/origin/*'
03:59:01.080152 read-cache.c:2297       performance: 0.000010826 s:  read cache .git/index
03:59:01.080455 run-command.c:663       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 'ssh -vvv' -o SendEnv=GIT_PROTOCOL git@<server> 'git-upload-pack '\''<repository>'\'''
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 4: Applying options for *
debug2: resolving "<server>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <server> [10.50.226.16] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /apps/home/jnknsprd/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version babeld-eb8eb2c
debug1: no match: babeld-eb8eb2c
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to <server>:22 as 'git'
debug3: hostkeys_foreach: reading file "/apps/home/jnknsprd/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/jnknsprd/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <server>
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1-etm@openssh.com,hmac-sha1
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-sha1-etm@openssh.com,hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:7J1hDzoAs0vF9KOR1KbzMijEBIF4f0P2cYVzIU9zDJs
debug3: hostkeys_foreach: reading file "/apps/home/jnknsprd/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/jnknsprd/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from <server>
debug3: hostkeys_foreach: reading file "/apps/home/jnknsprd/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /apps/home/jnknsprd/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 10.50.226.16
debug1: Host '<server>' is known and matches the ECDSA host key.
debug1: Found key in /apps/home/jnknsprd/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /apps/home/jnknsprd/.ssh/id_rsa ((nil))
debug2: key: /apps/home/jnknsprd/.ssh/id_dsa ((nil))
debug2: key: /apps/home/jnknsprd/.ssh/id_ecdsa ((nil))
debug2: key: /apps/home/jnknsprd/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /apps/home/jnknsprd/.ssh/id_rsa
debug3: no such identity: /apps/home/jnknsprd/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /apps/home/jnknsprd/.ssh/id_dsa
debug3: no such identity: /apps/home/jnknsprd/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /apps/home/jnknsprd/.ssh/id_ecdsa
debug3: no such identity: /apps/home/jnknsprd/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /apps/home/jnknsprd/.ssh/id_ed25519
debug3: no such identity: /apps/home/jnknsprd/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
03:59:01.153528 trace.c:487             performance: 0.074021155 s: git command: git fetch --tags --force --progress -- git@<repository>.git '+refs/heads/*:refs/remotes/origin/*'

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:618)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:847)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1216)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1294)
	at hudson.scm.SCM.checkout(SCM.java:540)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1215)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:645)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:517)
	at hudson.model.Run.execute(Run.java:1896)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
ERROR: Error cloning remote repo 'origin'
Finished: FAILURE

I don’t know how the gitplugin provides keys but the verbose output only seems to be trying keys in your home directory.

What if you provide the credential where you define the git repository information instead of the global setting? I didn’t even know there was a global ssh key setting

So after more troubleshooting and testing, I found out that is only that specific repository that’s causing the issue. But I tried cloning the repo from the Jenkins node manually and other machines, it works fine. But it seem to only have issue cloning through Git?

Update:
Hi @MarkEWaite and @halkeye, so like 5 more hours later, finally figured out what happened. For some reason a branch was created with a / as branch name, so it was named blah/somename, and I guess Jenkins Git didn’t like that, but figured it out. Apologies for all the trouble.