Failed to connect to git repository after upgrade to Jenkins 2.332.1 with Git Source Code Management

Upgraded Jenkins to 2.332.1 and a few plugins to try to stay up to date. Now getting the following error:

`Failed to connect to repository : Command “/usr/bin/git ls-remote -h git-repo-link HEAD” returned status code 128:
stdout:
stderr: ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.`

I have no problem cloning the repo from command line. I have tried deleting the workspace, downgrading a few of the plugins, adding the Wipe out repository & force clone in Additional Behaviors under Source Code Management, and a few others.

I can connect to my repo without problem from command line by cloning and/or running ssh -T git@repo-url

Credentials Plugin v2.6.1
Git Plugin v4.11.0
Git Client Plugin v3.11.0
GIT Server Plugin v1.10

Any help besides working through hours of manually downgrades is greatly appreciated

The 2.332.1 upgrade guide says:

Use the Plugin Manager to upgrade all plugins before and after upgrading to Jenkins 2.332.1. See the Guava upgrade blog post for more details

Credentials plugin 2.6.1 needs to be upgraded to the most recent version. Any other plugins in your installation also need to be updated to their most recent version.

All plugins are fully up to date. I do get this message with the Credentials plugin. I am on the latest stable jenkins version 2.332.1

Still having the same issue sadly. Do I need to downgrade the version of Jenkins that I am on?

I’m not aware of anything in the upgrade to 2.332.1 that would break ssh credentials, so long as you followed the instructions in the upgrade guide and updated all plugins.

Are you certain that the credential you’re using in the Jenkins job definition is the same credential that is being used in the case where command line git is able to clone the repository? Sometimes a user will have a working local configuration but the credential that was used in the Jenkins job no longer has permissions to perform the operations it performed before.

I have created new credentials in the UI to make sure the right private key was being used within the job and am still experiencing this problem. I am on CentOS 7.4.

I can revisit permissions. I can see that the jenkins user has ownership and access to the ssh directory and the private key within that directory that is used for connecting to our repo. Are there specific permissions that directory and private key file need in order for the jenkins user to successfully access them from a job?

All plugins are up to date … only message I have is the screenshot I posted above that relates to the Credentials plugin which makes it seem like it isnt actually offered for installation? Do you think downgrading that plugin may help?

I don’t know if that’s the cause, but CentOS 7.4 is a very old version of CentOS 7. The current version is CentOS 7.9.

I doubt that downgrading that plugin will change it. The message that is being displayed is an expected message because a version of the credentials plugin is available that requires a Jenkins version newer than the Jenkins version you are running. The following Jenkins blog post describes that message in more detail

You could certainly experiment with other versions of the credentials plugin, but my suspicion is that the problem is related to file or directory permissions or something similar, not to plugin versions.

Thank you for your help. I am going to keep working on debugging. What has been really odd is that when I ssh onto the host and change user to the jenkins user I can clone and connect to the repo no problem. The second I try to run ssh -T git@repo-url in an execute shell I get ssh_exchange_identification: Connection closed by remote host. I can also verify by running whoami in the execute shell that the jenkins user is running those commands.

Im unsure what the difference is between me running those commands as the jenkins user in command line and the jenkins user running those commands from the execute shell in a job.

I have created brand new ssh username and private key credentials using the Jenkins UI using a private key I know connects to my repo properly. When setting up my repo in a job under Source Code Management, Git refuses to connect to my repository properly.

Are there any logs that I can look at for the git plugin or other plugins that may be related to this? Otherwise, I am only trying to debug off of logs that are coming out of the failed job itself. This is really frustrating and is making me hesitate to use and ever upgrade Jenkins in the future :confused:

You may find this technique helpful

Thanks Mark. That didnt provide too much more information. Atleast not much that appears to be all that helpful unfortunately. I am beginning to consider opening a bug report with Jenkins for continued support working through this.

LOL … I opened a bug report and it looks like it auto assigned to you Mark… Apologies for the redundancy, I just dont know where else to turn for help working past this.

3 posts were split to a new topic: Timeout connecting to GitHub

This is still a huge problem for me and is stopping me from being able to update to the latest version of Jenkins. I took some time and spun up a fresh install of jenkins to do some testing.

Jenkins 2.375.3
Credentials plugin [1214.v1de940103927]

When I ssh into the fresh jenkins install and run:

sudo su - jenkins -s /bin/bash
whoami
jenkins

I can confirm that I am the jenkins user. I am able to run: git ls-remote ssh://git@repo-url/path/reponame/repo.git successfully and see all the different branches that exist in my code repo.

I am using ssh connection to connect to our git repo and can confirm permissions and files are in place:

ls -al /var/lib/jenkins/.ssh/
total 24
drwxrwxr-x.  2 jenkins jenkins   53 Mar  7 17:04 .
drwxr-xr-x. 16 jenkins jenkins 8192 Mar  7 17:42 ..
-rw-------.  1 jenkins jenkins  235 Mar  7 17:03 config
-r--------.  1 jenkins jenkins 1675 Mar  7 17:03 id_rsa
-rw-r--r--.  1 jenkins jenkins  399 Mar  7 17:04 known_hosts

At this point I am able to confirm that as the jenkins user… I can access the repository successfully.

Once I create a freestyle project and add ssh://git@repo-url/path/reponame/repo.git to the source code management repository URL I immediately get the same status code 128 error. I created global credentials using “SSH Username with private key” where I inputed the git username with the private key that I can confirm is working via terminal.

Here is full output of job failure:

Started by user My Name
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/ssh-connection
The recommended git tool is: NONE
using credential f13366cf-10bb-42ca-9548-806da50ebd0d
 > /bin/git rev-parse --resolve-git-dir /var/lib/jenkins/workspace/ssh-connection/.git # timeout=10
Fetching changes from the remote Git repository
 > /bin/git config remote.origin.url ssh://git@repo-url/path/reponame/repo.git # timeout=10
Fetching upstream changes from ssh://git@repo-url/path/reponame/repo.git
 > /bin/git --version # timeout=10
 > git --version # 'git version 2.39.2'
using GIT_SSH to set credentials 
Verifying host key using known hosts file
 > /bin/git fetch --tags --force --progress -- ssh://git@repo-url/path/reponame/repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from ssh://git@repo-url/path/reponame/repo.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1003)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1245)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1309)
	at hudson.scm.SCM.checkout(SCM.java:540)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1241)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)
	at hudson.model.Run.execute(Run.java:1900)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:107)
	at hudson.model.Executor.run(Executor.java:449)
Caused by: hudson.plugins.git.GitException: Command "/bin/git fetch --tags --force --progress -- ssh://git@repo-url/path/reponame/repo.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: 17:35:44.812309 git.c:460               trace: built-in: git fetch --tags --force --progress -- ssh://git@repo-url/path/reponame/repo.git '+refs/heads/*:refs/remotes/origin/*'
17:35:44.812533 read-cache.c:2481       performance: 0.000006996 s:  read cache .git/index
17:35:44.812718 run-command.c:655       trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 '"ssh -vvv"' -o SendEnv=GIT_PROTOCOL git@repo-url 'git-upload-pack '\''/path/reponame/repo.git'\'''
"ssh -vvv": ssh -vvv: command not found
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
17:35:44.814756 trace.c:411             performance: 0.002698970 s: git command: /bin/git fetch --tags --force --progress -- ssh://git@repo-url/path/reponame/repo.git'+refs/heads/*:refs/remotes/origin/*'

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2734)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2111)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:623)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:1001)
	... 11 more
ERROR: Error fetching remote repo 'origin'
Finished: FAILURE

I have confirmed that my private key and public key connection is working and correct.

For whatever reason I am still unable to get a jenkins job on the controller to successfully connect to my git repo after updating… or in this specific case… I cant even set it up on a fresh install as it seems jenkins jobs are unable to access my credentials properly.

Any continued support would be greatly appreciated… this problem is forcing me to consider different tools such as Github for our CI/CD pipelines.

Wild guesses of what might be wrong

The diagnostics message ssh -vvv: command not found suggests that your attempt to insert the diagnostics is preventing command line git from finding the ssh command that it will use for authentication. Since command line git cannot find ssh, it cannot authenticate.

Alternatives

  • Enable JGit and configure the job to use JGit instead of command line git
  • Switch to use https transport instead of ssh
  • Remove the diagnostic settings (environment variables, etc.) that were previously inserted

Since the output reports that you are running command line git 2.39.2, I assume you are no longer running CentOS 7. If you are still running CentOS 7, then be sure that your operating system is updated with the latest patches.

Hi,
I had the same issue and fixed it by resetting and removing agent host cached files. just moved to another location reconnected using same credentials