My ssh nodes have gone offline due to "The selected credentials cannot be found"

Hi,

Since updating to 2.322 on Friday (2021-11-26), all my Linux nodes are offline ;-(.

The log says:

java.lang.InterruptedException: Validate configuration:
The selected credentials cannot be found

When I try to add credentials for the node, I can’t see any listed.

My Windows nodes work fine. These are set up with “Launch agent by connecting it to the controller”. I could try that for the Linux nodes, but just wanted to be sure I’m not missing something obvious before I go about reworking my setup.

The credentials I expect to see here are stored globally and the (global) store is listed as scoped to Jenkins. When I try to view that detail, however, I get:
image

Has something got corrupted, I wonder?

If it’s any clue, I did find that “sudo service jenkins start” did not complete (timed out and gave “FAILED”) since v2.320 and in order to upgrade I had to forcefully kill the Jenkins process. I can imagine how that might have led to some corruption.

There is nothing I can spot that obviously relates to this problem in /var/log/jenkins/jenkins.log

Any tips or further diagnostics greatly appreciated.

Thank you,
Tim

UPDATE: Not all of my Linux nodes can get through the firewall to reach the Jenkins system ;-(

so it’s solved? I’m really not sure what problem you are trying to describe. There’s lots of information.

Are you saying you see a credential listed, but when you go to view the credential it says not found?

Assuming you are using the default credential store, all your credentials will be stored on disk, encrypted, inside of credentials.xml. Is the credential your trying to access in there (It’ll be credentialdid)

If you updated last week, did you also update all plugins? I don’t think credentials is one of them, but some plugins have to use new APIs as jenkins core had to update one of the internal guava libraries.

No, it’s not sorted.
I’ve adjusted 2 of my nodes to connect to Jenkins, rather than be launched by but 2-3 of them can’t connect that way around through the firewall, so won’t launch.

I am seeing credentials listed in “Manage credentials” and these are showing in credentials.xml. When I try to assign a credential to a node to launch the agent, it does not list any.

Yes, I updated all plugins, too.
There’s an update for Pipeline: Groovy which I have applied (pending restart).

I’ve updated to 2.323 and it’s not resolved the problem.

Further digging shows that it’s using the “SSH Build Agents” so I’m looking through the associated troubleshooting now.

I have reviewed the troubleshooting on the ssh build agents plugin and the JIRA issues logged for the ssh-credentials plugin and cannot see anything that would explain the problems.
Mostly, they indicate “authentication failed” issues, but my system isn’t even trying to authenticate.

The service startup still times out. journalctl -xe shows:

Dec 02 09:57:18 olibjenap01dxsh systemd[1]: jenkins.service start operation timed out. Terminating.
Dec 02 09:57:18 olibjenap01dxsh systemd[1]: Failed to start LSB: Jenkins Continuous Integration Server.
-- Subject: Unit jenkins.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit jenkins.service has failed.
--
-- The result is failed.
Dec 02 09:57:18 olibjenap01dxsh systemd[1]: jenkins.service failed.

UPDATE (problem remains and extent of problem is now identified as more serious):

I tried to build a project just now which uses separate credentials and this could not find those credentials. When I o to configure the project, it does not list them:
image

The console log advises:
13:26:50 ERROR: Could not find credentials entry with ID ‘71c0ccba-2ad8-400d-af62-8bbee39669bd’

I can find this ID in my credentials.xml file:

      <java.util.concurrent.CopyOnWriteArrayList>
        <com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
          <scope>GLOBAL</scope>
          <id>71c0ccba-2ad8-400d-af62-8bbee39669bd</id>
          <description>Tim Scott/SVN</description>
          <username>scottt</username>
          <password>{AQAAABAAAAAQtRjHZ (snip) NRYxqD/8LE=}</password>
          <usernameSecret>false</usernameSecret>
        </com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>

Any tips greatly appreciated.

Thanks,
Tim

UPDATE (potentially diagnostically significant?)
I thought I’d re-add the credentials in case it’s just a corrupted file somehow.

I can’t add credentials as the “kind” has no options :-(.
image

:cry:

UPDATE (working around slowly)
I wiped all the credentials and am re-entering them to try to get the nodes and projects working again.
I did notice that the Credential Providers list didn’t match with my expectations, so I’ve redone those, too.

Now I am getting more what I expect.
(One node just doesn’t connect, though. No messages - it just hangs … digging continues)