Host key verification for ec2 instances buggy?

Using Jenkins
1

I am having trouble with CHECK_NEW_HARD hostkey verification strategy… it seemed to work fine for me when testing on my dev server but in production with all the same versions and such… for two platforms: ubuntu 22 and 24.

After spawning the agent is disabled before the log shows that the server key is accepted and login happens.

e.g. I see this in the log:

Jan 28, 2026 9:44:21 PM hudson.plugins.ec2.EC2Cloud
INFO: Connection allowed after the host key has been verified

and then the UI says:

Jan 28, 2026, 9:43:49 PM
The instance SSH key was unexpected or impossible to check

smells like a bug. any thoughts?

I can bring the node back online manually but as-is I have an AWS instance spawned that is “wasted” aka not in use but costing $$ :slight_smile:

Jenkins setup:

Jenkins: 2.541.1
OS: Linux - 6.1.0-30-cloud-arm64
Java: 21.0.7 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
ant:520.vd082ecfb_16a_9
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.6-183.ve5a_8a_b_e71e59
asm-api:9.9.1-189.vb_5ef2964da_91
authentication-tokens:1.144.v5ff4a_5ec5c33
aws-credentials:254.v978a_5e206a_d7
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.33.4-62.vc1a_8df64b_4c9
aws-java-sdk2-ec2:2.33.4-62.vc1a_8df64b_4c9
badge:2.562.vc3c0b_6925377
bootstrap5-api:5.3.8-895.v4d0d8e47fea_d
bouncycastle-api:2.30.1.82-277.v70ca_0b_877184
branch-api:2.1268.v044a_87612da_8
build-blocker-plugin:171.ve0716a_f69636
build-timeout:1.39
built-on-column:1.5
caffeine-api:3.2.3-194.v31a_b_f7a_b_5a_81
checks-api:402.vca_263b_f200e3
clone-workspace-scm:121.vf1f73b_d671ef
cloud-stats:377.vd8a_6c953e98e
cloudbees-folder:6.1073.va_7888eb_dd514
command-launcher:123.v37cfdc92ef67
commons-httpclient3-api:3.1-3
commons-lang3-api:3.20.0-109.ve43756e2d2b_4
commons-text-api:1.15.0-210.v7480a_da_70b_9e
conditional-buildstep:1.5.0
config-file-provider:1006.vc7366c201f57
configuration-as-code:2036.v0b_c2de701dcb_
configuration-as-code-groovy:1.1
copyartifact:795.ve8e151429b_27
credentials:1480.v2246fd131e83
credentials-binding:702.vfe613e537e88
cvs:502.v02d5a_0c6b_9a_3
dark-theme:574.va_19f05d54df5
dashboard-view:2.558.v96b_901978e47
data-tables-api:2.3.5-1497.v38449eb_7d5a_1
declarative-pipeline-migration-assistant:1.6.6
declarative-pipeline-migration-assistant-api:1.6.6
display-url-api:2.217.va_6b_de84cc74b_
docker-commons:457.v0f62a_94f11a_3
docker-workflow:634.vedc7242b_eda_7
dtkit-api:3.0.3
durable-task:651.v1f5e074fc83f
ec2:2045.v06da_da_a_46422
echarts-api:6.0.0-1165.vd1283a_3e37d4
eddsa-api:0.3.0.1-29.v67e9a_1c969b_b_
email-ext:1933.v45cec755423f
embeddable-build-status:637.vd878e68178f8
emoji-symbols-api:17.0-57.v8d44b_9a_b_d5ea_
envinject:2.934.vc674e76cf954
envinject-api:1.237.v82803a_511906
extended-read-permission:68.vd270568a_7520
external-monitor-job:223.vb_fddcf42c9b_3
ez-templates:1.3.5
font-awesome-api:7.1.0-882.v1dfb_771e3278
git:5.9.0
git-client:6.5.0
git-server:137.ve0060b_432302
github:1.45.0
github-api:1.330-492.v3941a_032db_2a_
github-branch-source:1917.v9ee8a_39b_3d0d
gradle:2.18.1203.v2c96b_1243c72
groovy:497.v7b_061a_a_de65d
groovy-postbuild:300.va_253b_2988cb_1
gson-api:2.13.2-173.va_a_092315913c
htmlpublisher:427
http_request:1.24
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:94.vcc3065403257
jackson2-api:2.20.1-423.v13951f6b_6532
jakarta-activation-api:2.1.4-1
jakarta-mail-api:2.1.5-1
jakarta-xml-bind-api:4.0.6-12.vb_1833c1231d3
javadoc:354.vee1a_660b_4990
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-143.v5979df3304e6
jclouds-jenkins:2.43
jdk-tool:83.v417146707a_3d
jenkins-multijob-plugin:662.vd2e0001f6b_b_d
jjwt-api:0.11.5-120.v0268cf544b_89
jnr-posix-api:3.1.21-186.vb_7ec9b_23ce83
job-dsl:1.93
joda-time-api:2.14.0-177.vd7e9347b_e7d5
jquery:1.12.4-3
jquery3-api:3.7.1-619.vdb_10e002501a_
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20251224-185.v0cc18490c62c
json-path-api:2.10.0-202.va_9cc16c1e476
jsoup:1.22.1-76.v9cdb_2456c0e3
junit:1369.v15da_00283f06
junit-attachments:330.v25180b_263160
ldap:807.v7d7de30930cf
lockable-resources:1438.v3c0f8c9e2060
mailer:525.v2458b_d8a_1a_71
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.9
matrix-combinations-parameter:1.3.3
matrix-project:870.v9db_fcfc2f45b_
maven-plugin:3.27
mina-sshd-api-common:2.16.0-167.va_269f38cc024
mina-sshd-api-core:2.16.0-167.va_269f38cc024
mina-sshd-api-scp:2.16.0-167.va_269f38cc024
miniorange-saml-sp:2.4.9
monitoring:2.6.0
multiple-scms:0.8
naginator:1.530.vb_6d120f250b_1
next-build-number:66.v4b_4762172d53
node-iterator-api:72.vc90e81737df1
nodelabelparameter:851.vd94e5048d321
oauth-credentials:0.657.v7d8dd90b_0382
okhttp-api:4.12.0-195.vc02552c04ffd
oss-symbols-api:442.v99039087229b_
pam-auth:1.12
parameterized-trigger:873.v8b_e37dd8418f
pipeline-build-step:584.vdb_a_2cc3a_d07a_
pipeline-github:2.8-162.382498405fdc
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:245.v88f03631a_b_21
pipeline-groovy-lib:787.ve2fef0efdca_6
pipeline-input-step:540.v14b_100d754dd
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2277.v00573e73ddf1
pipeline-model-definition:2.2277.v00573e73ddf1
pipeline-model-extensions:2.2277.v00573e73ddf1
pipeline-rest-api:2.39
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2277.v00573e73ddf1
pipeline-stage-view:2.39
plain-credentials:199.v9f8e1f741799
plugin-util-api:6.1192.v30fe6e2837ff
postbuild-task:78.v24529f1f5cdb_
powershell:2.3
preSCMbuildstep:115.vf9808414429d
prism-api:1.30.0-703.v116fb_3b_5b_b_a_a_
promoted-builds:992.va_00888f21b_74
publish-over:0.22
publish-over-ssh:390.vb_f56e7405751
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
run-condition:276.v97298f3a_cd51
saferestart:102.v4dc1b_9636a_ee
saml:4.595.vec7523b_5d543
scm-api:724.v7d839074eb_5c
script-security:1385.v7d2d9ec4d909
scripted-cloud-plugin:0.12
slack:795.v4b_9705b_e6d47
snakeyaml-api:2.5-143.v93b_c004f89de
ssh-agent:386.v36cc0c7582f0
ssh-credentials:361.vb_f6760818e8c
ssh-slaves:3.1096.v0b_cc466e4323
sshd:3.374.v19b_d59ce6610
structs:362.va_b_695ef4fdf9
subversion:1303.vcfd9679fb_c12
theme-manager:327.v780d7096ec29
timestamper:1.30
token-macro:477.vd4f0dc3cb_cf1
trilead-api:2.284.v1974ea_324382
variant:70.va_d9f17f859e0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1384.vdc05a_48f535f
workflow-basic-steps:1098.v808b_fd7f8cf4
workflow-cps:4254.v0c8e228524ea_
workflow-durable-task-step:1464.v2d3f5c68f84c
workflow-job:1571.vb_423c255d6d9
workflow-multibranch:821.vc3b_4ea_780798
workflow-scm-step:466.va_d69e602552b_
workflow-step-api:710.v3e456cc85233
workflow-support:1010.vb_b_39488a_9841
ws-cleanup:0.49
xml-job-to-job-dsl:0.1.13
2

We’ve seen the same issue on ci.jenkins.io. We reported it as a plugin bug at:

3

Thanks. I posted some of my details there. For me I think I can switch to CHECK_NEW_SOFT for hostKeyVerificationStrategy and put this on the back-burner somewhat.

4

sorry for hijacking this, but how did you even get to the point of “INFO: Connection allowed after the host key has been verified”?

For me it always says “didn’t print the host key. Expected a line starting with: “ecdsa-sha2-nistp256”. Even though i can see the line when i retrieve the output myself. Did you have any problems like that?

5

@sebastianreloaded no worries… I did have to figure out WHERE to cat out the public sshd keys for the “console” and in aws ec2 case (maybe ec2 plugin especially) this means /dev/console. I can’t find the reference for /dev/console but generally here is the AWS info about the get-console-output that the ec2 plugin uses I think (from reading the plugin java code).

So I put in my userData in configuration as code ec2 template data:

cat /etc/ssh/*.pub | tee -a /dev/console # print out public ssh hostkeys so that jenkins hostKeyVerificationStrategy CHECK_NEW_HARD works
1 Like
6

@craigcomstock thank you! this is exactly what i read and using get-console-output i see all the entries from /etc/ssh/*.pub in the console output already, BUT by default this comes at the very end of the output. Maybe if i force it to be earlier it will work.
i’ll try, thanks again