Help needed on jenkins for ssh credentials (docker and k8s) setup

Using Jenkins
1

Background & The Problem

I have not touch Jenkins for the longest time. Been using other CI tools such as GitLab, GitHub actions, Drone for my organisational needs.

Recently am trying to get familiar back with Jenkins and so I started out with my home lab

Steps and Issue

The following is what I have setup for my homelab

services:
  jenkins:
    image: jenkins/jenkins:lts
    container_name: jenkins
    ports:
      - "8080:8080"       # Jenkins web interface
      - "50000:50000"     # For inbound Jenkins agents
    volumes:
      - jenkins_home:/var/jenkins_home
    restart: unless-stopped

volumes:
  jenkins_home:
    driver: local

Setting up credential steps and Issue

  1. I generate a pair of ssh keys using ssh-keygen -t ed25519 -C "me@myemail.com"

  2. Push the public key (~/.ssh/id_ed25519.jenkins.pub) to my gitlab settings.

  3. Test the key. GIT_SSH_COMMAND='ssh -i ~/.ssh/id_ed25519.jenkins' git clone git@gitlab.com:<path_to>/myrepo.git and it works.

  4. Go to Dashboard (http://localhost:8080), Manage Jenkins, Credentials.

  5. Under System, Global domain, Add credentials.

  6. Under Scope: Global, Id: git-jenkins, description: git-jenkins, username: me@myemail.com (This if followed this video. I also tried git).

  7. Copy and Paste my private key (~/.ssh/id_ed25519.jenkins). I tried both on my linux and windows machine. Save.

  8. When create a freestyle project, at the SCM section, I paste in the same git URI which I tested in step 3, and select the credentials from Step 7.

Failed to connect to repository : Command "git ls-remote -h -- git@gitlab.com:<path to>/myrepo.git HEAD" returned status code 128:
stdout:
stderr: git@gitlab.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Summary

I tried the videos, guides over the internet and ChatGPT. Nothing works.
As I explained I tried other machines (window arm and linux x86). And even fall back to an slightly older version of LTS. At this point, I am not sure what else am I missing?

I have tested by going into the container and test. It is not the issue of the known host file. As long as I mount the same key and use it. It works. I am pretty ascertain that there is something wrong with credentials Jenkin using SSH. or something else I might have missed.

Update

11/12/2024

In my script console, I did the following

import jenkins.model.*
import hudson.util.Secret
import com.cloudbees.plugins.credentials.*
import com.cloudbees.plugins.credentials.domains.*

def credentialsStore = Jenkins.instance.getExtensionList('com.cloudbees.plugins.credentials.SystemCredentialsProvider')[0]?.store

def providedPrivateKey = """
-----BEGIN OPENSSH PRIVATE KEY-----
MyPrivateKeyContentHere
-----END OPENSSH PRIVATE KEY-----
""".trim()
  
println "Scanning Global Domain in System Store..."

credentialsStore?.getCredentials(Domain.global())?.each { cred ->
    println "ID: ${cred.id}"
    println "Description: ${cred.description}"
    
    if (cred instanceof com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey) {
        println "Username: ${cred.username}"
        println "Private Key: ${cred.privateKey}"
        println "Passphrase: ${Secret.toString(cred.passphrase)}"
      	
      	def storedPrivateKey = cred.privateKey?.trim()
        
        if (storedPrivateKey == providedPrivateKey) {
            println "Match found for Credential ID: ${cred.id}"
        } else {
            println "No match for Credential ID: ${cred.id}"
        }
    }
    println "-------------------------"
}

and my result

Scanning Global Domain in System Store...
ID: jenkins
Description: jenkins
Username: git
Private Key: -----BEGIN OPENSSH PRIVATE KEY-----
MyPrivateKeyContentHere
-----END OPENSSH PRIVATE KEY-----

Passphrase: 
Match found for Credential ID: jenkins
-------------------------
Result: [com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey@99ee54b6]

The key in my credentials store matches the one I try to compare with. So now I ascertain the key are indeed correct but somehow the keys are not used properly from the store.

2

For step 6, what type of credential did you create?

3

Okay, watched the video with focus on the SSH section. If you followed that closely, you used an SSH credential.

Note that Jenkins is trying git ls-remote to verify the repository exists. Can you try that command locally on your system…
GIT_SSH_COMMAND='ssh -i ~/.ssh/id_ed25519.jenkins' git ls-remote git@gitlab.com:<path to>/myrepo.git

4

Yes been there done that. See step 3. I also clone my repo out.

→ GIT_SSH_COMMAND="ssh -i ~/.ssh/id_ed25519.jenkins" git ls-remote -h -- git@gitlab.com:42dev_pub/apps-pub/helloworld.git
417fe4b2950a0bc202304a89c6fcd84c1c61f073        refs/heads/main

You see I place the SSH private key in the /var/jenkins_home/.ssh , then it works. This will not even require me to select credentials.

So my point is, these seems to be reproducible issue yet seems like I am the only one facing it.

And to answer your question step 7 already show it is a ed25519 type key.

5

I’ve always used git as the username for GitLab ssh private key credentials. I think that you should try again with git as the username.

6

Step 6. I did mentioned I also tried using git as user name, saw other examples, the username does not really matters.