Ssh-agent Cannot pull from GitHub?

Hi there,

Just got most of our Jenkins setup working but I’m stuck on something I’m not sure how to fix and can’t seemingly find a solution for. We are self hosting the Jenkins box and using Docker (via cloud config) to run jobs in the ssh-agent:latest image. The issue we run into is when the agent tries to pull our code from GitHub we get the following error:

hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- https://github.com/***/***.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: fatal: unable to access 'https://github.com/***/***.git/': server certificate verification failed. CAfile: none CRLfile: none

I’m not sure how to fix this and can’t figure out the right string to put into Google to get a starting point answer. Am I just missing something simple? At this point I’m just not sure what to even try. I had tried adding the ca-certificates package in the build pipeline, but that didn’t even seem to run (we use a Jenkinsfile in the code so that likely makes sense).

Any help on this is appreciated.

My thinking is that github uses digicert, which has been trusted for a while. And the ssh-agent image is pretty recent. So unlikely its a cert issue.

My guess is either a) Your clocks are way off and ssl is complaining, but I think thats a different message. or b) You are actually failing to talk to a proxy that has a untrusted ssl cert. That would need to update your ca-cert to handle that.

That being said, why not use ssh instead of https for github?

Good catch on the method of fetching, I didn’t set up the job originally and I didn’t notice that. Swapping it to SSH it does produce a different error:

hudson.plugins.git.GitException: Command "git fetch --tags --force --progress -- git@github.com:****/****.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout: 
stderr: Host key verification failed.
fatal: Could not read from remote repository.

The authentication is via a GitHub app installed on the Org. The GitHub app’s test button returns that it’s all happy so I assumed that would be fine.

As for something like clocks being off, I’m pretty sure you are right that it’s a more explicit error pointing at time being off, so I don’t think it’s that. Each agent is spawned as a Docker container as the job is requested (via the Docker plugin).

"No ECDSA host key" error connecting to GitHub - #5 by koranke just had a bunch of useful items about that.

that runs on the controller, while this is running on your agent.

I’m not sure I understand how to run said command on dynamically spawned Docker containers. I can’t see anywhere that allows you to configure the ability to have something run inside the container before it tries to pull the git repo.

I had figured that by setting it as the credentials for the job it would inject the correct keys and such needed to access the repo? Am I misunderstanding what you mean maybe?
image

So i kinda skimmed and jumped to conclusions sorry. Re-reading I see you are running an ssh agent inside of docker, not specifically docker agents (different idea).

When you say dynamic agents, do you mean you are running ssh-agent docker image on a server, or do you mean jenkins is spinning up an agent on demand, and then running ssh-agent? I’ve not heard of on demand agents with the ssh-agent image before.

So looks like the powershell startup scripts (windows) have https://github.com/jenkinsci/docker-ssh-agent/blob/0c1d128c33d2d7c0fabb34ebbccb29a6d9c4fcf6/setup-sshd.ps1#L76-L82 but I don’t see the same behavior in the bash versions, which is probably why its not really advertised. I poked around on the scripts and couldn’t find anything that would pre-populate a known hosts file.

The easiest way would be to extend the docker image yourself

FROM jenkins/ssh-agent
RUN ssh-keyscan github.com > /etc/ssh/ssh_known_hosts

I would also recommend creating a github issue on that repo asking for it to be baked in, or customizable or something.

So we have the Docker plugin installed. What that allows us to do is use Docker to create ssh-agent agent containers on demand as “Cloud Agents”. We haven’t modified the container at all at this point so that’s likely what we will need to do.

Though it does seem like there is maybe something else at play as well. I swapped from using the GitHub plugin’s GitHub App integration as the “credentials” for the source management in the job config to using a SSH key created as my account and it’s happy, so I think there’s something missing there that’s either not clearly documented or I missed somewhere. While this is working in the short term, we didn’t want it tied to any one person’s account and rather liked the idea of it being tied directly to the org via a GitHub App. So I’ll need to try and figure that our at some point soon, but at least it seems to be mostly working for now to evaluate and test our new test suite.