Ansible plugin - Failed to connect to the host via ssh: Host key verification failed

Using Jenkins
1

Jenkins setup:

Jenkins: 2.492.2
OS: Linux - 6.8.0-57-generic
Java: 21.0.6 - Ubuntu (OpenJDK 64-Bit Server VM)

ansible:524.v9fa_a_4c989224
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.7.1-97.v4cc844130d97
bootstrap5-api:5.3.3-2
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1214.v3f652804588d
build-pipeline-plugin:2.0.2
caffeine-api:3.2.0-161.v691ef352cee1
checks-api:367.v18b_7f530e54a_
cloudbees-folder:6.985.va_f1635030cc5
command-launcher:118.v72741845c17a_
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-153.v91dcd89e2a_22
credentials:1408.va_622a_b_f5b_1b_1
credentials-binding:687.v619cb_15e923f
display-url-api:2.209.v582ed814ff2f
durable-task:587.v84b_877235b_45
echarts-api:5.6.0-2
eddsa-api:0.3.0-13.v7cb_69ed68f00
font-awesome-api:6.7.2-1
gson-api:2.12.1-113.v347686d6729f
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:82.v0597178874e1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
joda-time-api:2.13.1-115.va_6b_5f8efb_1d8
jquery:1.12.4-3
jquery3-api:3.7.1-3
json-api:20250107-125.v28b_a_ffa_eb_f01
junit:1317.v5b_35d792b_06a_
mailer:489.vd4b_25144138f
matrix-auth:3.2.5
matrix-project:845.vffd7fa_f27555
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
parameterized-trigger:840.v3c7d4a_a_5e6c7
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-model-api:2.2247.va_423189a_7dff
pipeline-model-definition:2.2247.va_423189a_7dff
pipeline-model-extensions:2.2247.va_423189a_7dff
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2247.va_423189a_7dff
pipeline-timeline:1.0.3
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:6.0.0
scm-api:704.v3ce5c542825a_
script-security:1373.vb_b_4a_a_c26fa_00
snakeyaml-api:2.3-123.v13484c65210a_
ssh-credentials:355.v9b_e5b_cde5003
sshd:3.353.v2b_d33c46e970
structs:343.vdcf37b_a_c81d5
trilead-api:2.192.vc50a_d147e369
variant:70.va_d9f17f859e0
workflow-api:1366.vf1fb_e1a_f6b_22
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4045.v0efb_cb_7cea_e9
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1505.vea_4b_20a_4a_495
workflow-multibranch:803.v08103b_87c280
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:963.va_600813d04a_a_

Hello everyone,
I’m trying to get Jenkins’ ansible plugin to work, making it execute a playbook on a remote target (IP ..*.51), but I have issues with SSH authentication.

Here is pipeline’s script:

pipeline {
    agent any

    stages {
        stage('run CIS playbook') {
            steps {
ansiblePlaybook credentialsId: 'UID_AND_PRIVATE_KEY', installation: 'Ansible first', inventory: '/etc/ansible/hosts', playbook: '/opt/ansible/UBUNTU22-CIS-devel/site.yml', vaultTmpPath: ''            }
        }
    }
}

As you can see from above script, the SSH remote user’s credentials (name and SSH private key) are saved into Jenkins’ credentials storage, under ID ‘UID_AND_PRIVATE_KEY’.
Unfortunately, running the pipeline gives this output to console (file /etc/ansible/hosts only lists target host ..*.51 in this testbed):

Started by user admin
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/ansible lockdown no.4 pipe
[Pipeline] {
[Pipeline] stage
[Pipeline] { (run CIS playbook)
[Pipeline] ansiblePlaybook
[ansible lockdown no.4 pipe] $ /usr/bin/ansible-playbook /opt/ansible/UBUNTU22-CIS-devel/site.yml -i /etc/ansible/hosts --private-key "/var/lib/jenkins/workspace/ansible lockdown no.4 pipe/ssh9863544424216943852.key" -u m***f

PLAY [Apply ansible-lockdown hardening] ****************************************

TASK [Gathering Facts] *********************************************************
fatal: [*.*.*.51]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.", "unreachable": true}

PLAY RECAP *********************************************************************
*.*.*.51                 : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

FATAL: command execution failed
hudson.AbortException: Ansible playbook execution failed
	at PluginClassLoader for ansible//org.jenkinsci.plugins.ansible.AnsiblePlaybookBuilder.perform(AnsiblePlaybookBuilder.java:328)
	at PluginClassLoader for ansible//org.jenkinsci.plugins.ansible.workflow.AnsiblePlaybookStep$AnsiblePlaybookExecution.run(AnsiblePlaybookStep.java:466)
	at PluginClassLoader for ansible//org.jenkinsci.plugins.ansible.workflow.AnsiblePlaybookStep$AnsiblePlaybookExecution.run(AnsiblePlaybookStep.java:365)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate2(ACL.java:451)
	at PluginClassLoader for workflow-step-api//org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: Ansible playbook execution failed
Finished: FAILURE

So the issue is ‘Host key verification failed’.To overcome this, I tried to disable verification of the host key, adding hostKeyChecking switch in the pipeline’s script code as in:

ansiblePlaybook credentialsId: 'UID_AND_PRIVATE_KEY', hostKeyChecking: 'N', installation: 'Ansible first', inventory: '/etc/ansible/hosts', playbook: '/opt/ansible/UBUNTU22-CIS-devel/site.yml', vaultTmpPath: ''            }

but to no avail (nor did hostKeyChecking: ‘NO’ work either). The hostKeyChecking switch is listed in the ansible plugin documentation page at Ansible where it reads that it toggles checking of the host key by setting the environment variable ANSIBLE_HOST_KEY_CHECKING, but I couldn’t find such environment variable listed under Dashboard > Manage Jenkins > System Information > Environment Variables…

I stress out that this is a Jenkins issue, because I am perfectly able to login to host ..*.51 via SSH from terminal on the same box where Jenkins runs, and SSH auth of ansible’s CLI also works nice and easy when I type from terminal (always on same Jenkins box) this command:

/usr/bin/ansible-playbook /opt/ansible/UBUNTU22-CIS-devel/site.yml -i /etc/ansible/hosts --private-key ~/.ssh/id_rsa --become-user m***f -C

I don’t know if the problem is with known_hosts file, that has the following permissions:

-rw-r--r-- 1 user01 user01 1262 apr 11 18:55 known_hosts

sshd log on target host ..*.51 is as follows:

Apr 12 11:03:36 ubu-01 sshd[1675]: debug1: Forked child 1691.
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: Set /proc/self/oom_score_adj to 0
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: rexec start in 4 out 4 newsock 4 pipe 6 sock 7
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: inetd sockets after dupping: 4, 4
Apr 12 11:03:36 ubu-01 sshd[1691]: Connection from *.*.*.100 port 49092 on *.*.*.51 port 22 rdomain ""
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.11
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.9p1 Ubuntu-3ubuntu0.11
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: compat_banner: match: OpenSSH_8.9p1 Ubuntu-3ubuntu0.11 pat OpenSSH* compat 0x04000000
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: permanently_set_uid: 113/65534 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: SSH2_MSG_KEXINIT received [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: kex: algorithm: ecdh-sha2-nistp256 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: zlib@openssh.com [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: zlib@openssh.com [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: SSH2_MSG_KEX_ECDH_INIT received [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: ssh_packet_send2_wrapped: resetting send seqnr 3 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: rekey out after 4294967296 blocks [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: Connection closed by *.*.*.100 port 49092 [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: do_cleanup [preauth]
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: monitor_read_log: child log fd closed
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: do_cleanup
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: Killing privsep child 1693
Apr 12 11:03:36 ubu-01 sshd[1691]: debug1: audit_event: unhandled event 12

so it is Jenkins’ box ..*.100 that closes the connection to target 51 for some reason:

Apr 12 11:03:36 ubu-01 sshd[1691]: Connection closed by *.*.*.100 port 49092 [preauth]

Both Jenkins box 100 and target host 51 are Ubuntu 22.04 Virtual Machines (64 bit).
Jenkins version: 2.492.2
Ansible plugin version: 524.v9fa_a_4c989224
Java version: 21.0.6
Jenkins’ user.name: jenkins
java.vm.name: OpenJDK 64-Bit Server VM
java.vm.vendor: Ubuntu
browser: Firefox 137.0.1 (64 bit)

I’m stuck here and don’t have a clue how to fix this.
Any help welcome.

2

is the user user01 the user as which Jenkins is running? When you installed Jenkins as a systemd service it normally runs as user jenkins.
According to the ansible documentation you would need to set ANSIBLE_HOST_KEY_CHECKING=False to. So try setting

ansiblePlaybook credentialsId: 'UID_AND_PRIVATE_KEY', hostKeyChecking: 'False', installation: 'Ansible first', inventory: '/etc/ansible/hosts', playbook: '/opt/ansible/UBUNTU22-CIS-devel/site.yml', vaultTmpPath: ''            }

if you aren’t able to get it working via known_hosts file

3

Thanks for your kind reply, @mawinter69, please find below my observations.

no, Jenkins service runs as ‘jenkins’ user, as in the default setup;

I tried both: I set ANSIBLE_HOST_KEY_CHECKING=“False” in /etc/environment file and also added hostKeyChecking: 'False' to the pipeline’s script code. Then I rebooted the OS to have all users load the new environment variable. Neither one solution worked (I also tried to put both of them in place at the same time), the output error is exactly the same as before:

fatal: [*.*.*.51]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.", "unreachable": true}

Maybe worth noting that:

  1. the console output doesn’t show any hostKeyChecking: ‘False’ switch when executing the pipeline’s ansible command. It indeed reads exactly as before I added that switch; [ansible lockdown no.4 pipe] $ /usr/bin/ansible-playbook /opt/ansible/UBUNTU22-CIS-devel/site.yml -i /etc/ansible/hosts --private-key "/var/lib/jenkins/workspace/ansible lockdown no.4 pipe/ssh9863544424216943852.key" -u m***f
  2. ANSIBLE_HOST_KEY_CHECKING doesn’t show up at all in Dashboard → Manage Jenkins → System Information → Environment Variables
  3. As before, I am perfectly able to run the ansible playbook via CLI/SSH to scan target host ..*.51 I mean to say that when I launch this command from command line:
    /usr/bin/ansible-playbook /opt/ansible/UBUNTU22-CIS-devel/site.yml -i /etc/ansible/hosts --private-key ~/.ssh/id_rsa --become-user m***f -C it works smooth and nice…
4
  1. likely that the plugin is not showing this in the log as it sets an env variable at process level during process creation
  2. Dashboard → Manage Jenkins → System Information → Environment Variables it will not show up there because what you define in /etc/environment is only loaded in interactive shells I think. Jenkins is not started this way when you installed it as a service. You would need to add it to the systemd unit file of the Jenkins service
  3. Try to run this command as user jenkins, this should result in the host key getting added

Looking at the code it seems that hostKeyChecking is deprecated and does nothing anymore.
Try with disableHostKeyChecking: true

5

wow, not yet fixed, but improving! Many thanks!
After using switch disableHostKeyChecking: 'True' in the pipeline’s script, it seems that finally target host *.*.*.51 was added to the list of known hosts!

fatal: [*.*.*.51]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Warning: Permanently added '*.*.*.51' (ED25519) to the list of known hosts.\r\nAuthorized uses only. All activity may be monitored and reported.\nLoad key \"/var/lib/jenkins/workspace/ansible lockdown no.4 pipe/ssh16565410980015576964.key\": error in libcrypto\r\nm***f@*.*.*.51: Permission denied (publickey,password).", "unreachable": true}

Anyway user m***f (the user whose SSH credentials are supposed to be used by the plugin to log in to remote target 51, as per switch credentialsId: 'UID_AND_PRIVATE_KEY) is still denied permission to access:

error in libcrypto\r\nm***f@*.*.*.51: Permission denied (publickey,password).", "unreachable": true}

How do I go on now?

6

Obviously something is wrong with the ssh key you stored in Jenkins as a credential with id UID_AND_PRIVATE_KEY. Could be the key in that credential or the username that is wrong

7

yes, it turned out that while copying-and-pasting the ssh key I omitted the —BEGIN PRIVATE KEY— and —END PRIVATE KEY— header and footer. Thanks.

8

as a little debrief:

  • disableHostKeyChecking: ‘True’ was not needed
  • after having correctly copied id_rsa private key into jenkins credentials, ssh connection still fails with ‘missing sudo password’ error: I had to hard-copy it into inventory file /etc/ansible/hosts
  • there are two distinct ansible plugin reference pages, one at Ansible , the other at Ansible plugin : couldn’t find time to see if the content is consistent
  • nowhere was I able to find out that hostKeyChecking is deprecated in neither of the above two reference pages (thanks again @mawinter69). They should maybe be updated, if I am right.
  • still finding discrepancies between running ansible-playbook from command line (working) and from jenkins’ pipeline (errors). For example, switch --skip-tags “tag1, tag2, tag3” works as expected from command line, while skippedTags: ‘tag1, tag2, tag3’ doesn’t work at all in pipeline’s script (though it shows up in console’s output). Maybe I will start a separate topic on this issue…