ERROR: Cannot resume build because FlowNode 57 for FlowHead 1 could not be loaded

Using Jenkins Ask a question
1

I’m running into a weird error that I cant explain. I have a specific project which for which builds first finish successfully, and then afterwards get marked as Failed. The console output for that build ends with something as following:

[Pipeline] End of Pipeline
[Bitbucket] Notifying commit build result
[Bitbucket] Build result notified
Finished: SUCCESS
Creating placeholder flownodes because failed loading originals.
ERROR: Cannot resume build because FlowNode 57 for FlowHead 1 could not be loaded.
[Bitbucket] Notifying commit build result
[Bitbucket] Build result notified
Finished: FAILURE

The time between the ‘SUCCES’ and failure can vary greatly, sometimes as short as 40minutes sometimes up to 12 hours. I have looked at the post here:
ERROR: Cannot resume build because FlowNode 19

But my builds don’t show the additional part related to the performance setting. Also our Jenkins is configured to use the pipeline default (MAX_SURVIVABILITY) already. I have even tried to add the setting manually to our JenkinsFile, but that made no difference.

On the project I have had to ‘kill’ jobs at some point. Could it be that there are some historic remnants of these builds still lingering somewhere that could result in this issue?

No other jobs/projects in our controller experience this issue.

As additional information to the jenkins setup below. This job is the only job which is assigned to an inbound agent (running windows 11, remoting version 3206.vb_15dcf73f6a_9, JVM version 21.0.1+12-LTS-29 both identical to what our controller is using).

I cant make much sense of the logs but something that keeps popping up that is suspicious is:

May 03, 2024 7:59:34 AM WARNING org.jenkinsci.plugins.workflow.cps.CpsFlowExecution notifyListeners

null
java.lang.IllegalStateException: Can't send a remote FilePath to a different remote channel (current=hudson.remoting.Channel@5d28f3a7:JNLP4-connect connection from <redacted>, target=null)
	at hudson.FilePath.writeObject(FilePath.java:3554)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.thoughtworks.xstream.core.util.SerializationMembers.callWriteObject(SerializationMembers.java:154)
Caused: com.thoughtworks.xstream.converters.ConversionException: Failed calling method
---- Debugging information ----
message             : Failed calling method
cause-exception     : java.lang.IllegalStateException
cause-message       : Can't send a remote FilePath to a different remote channel (current=hudson.remoting.Channel@5d28f3a7:JNLP4-connect connection from <redacted>, target=null)
method              : hudson.FilePath.writeObject()
-------------------------------
	at com.thoughtworks.xstream.core.util.SerializationMembers.callWriteObject(SerializationMembers.java:161)

which maybe seems related? Can anyone help me get my pipeline back to a functioning state?

Jenkins setup:
Jenkins: 2.440.3
OS: Windows 10 - 10.0
Java: 21.0.1 - Oracle Corporation (Java HotSpot™ 64-Bit Server VM)

analysis-model-api:12.3.3
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.53.v1c90fd9191a_b_
basic-branch-build-strategies:81.v05e333931c7d
bitbucket:241.v6d24a_57f9359
bitbucket-build-status-notifier:1.4.2
blueocean:1.27.12
blueocean-bitbucket-pipeline:1.27.12
blueocean-commons:1.27.12
blueocean-config:1.27.12
blueocean-core-js:1.27.12
blueocean-dashboard:1.27.12
blueocean-display-url:2.4.2
blueocean-events:1.27.12
blueocean-git-pipeline:1.27.12
blueocean-github-pipeline:1.27.12
blueocean-i18n:1.27.12
blueocean-jwt:1.27.12
blueocean-personalization:1.27.12
blueocean-pipeline-api-impl:1.27.12
blueocean-pipeline-editor:1.27.12
blueocean-pipeline-scm-api:1.27.12
blueocean-rest:1.27.12
blueocean-rest-impl:1.27.12
blueocean-web:1.27.12
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1163.va_f1064e4a_a_f3
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-bitbucket-branch-source:883.v041fa_695e9c2
cloudbees-folder:6.901.vb_4c7a_da_75da_3
clover:4.14.2.596.vb_4d6475e990b_
cobertura:1.17
code-coverage-api:4.99.0
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-109.vfe16c66636eb_
conditional-buildstep:1.4.3
copyartifact:722.v0662a_9b_e22a_c
coverage:1.14.0
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:657.v2b_19db_7d6e6d
custom-folder-icon:2.12
dark-theme:439.vdef09f81f85e
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.5-1
display-url-api:2.200.vb_9327d658781
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.4-86.v39b_a_5ede342c
docker-plugin:1.6
docker-workflow:572.v950f58993843
doclinks:0.7
dtkit-api:3.0.2
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
email-ext:1806.v856a_01a_fa_39a_
embeddable-build-status:487.va_0ef04c898a_2
external-monitor-job:215.v2e88e894db_f8
favorite:2.208.v91d65b_7792a_c
font-awesome-api:6.5.2-1
forensics-api:2.4.0
git:5.2.1
git-client:4.7.0
git-forensics:2.1.0
git-parameter:0.9.19
github:1.38.0
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1787.v8b_8cd49a_f8f1
gradle:2.11
gson-api:2.10.1-15.v0d99f670e0a_7
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
htmlpublisher:1.33
instance-identity:185.v303dc7c645f9
ionicons-api:70.v2959a_b_74e3cf
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-design-language:1.27.12
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.87
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
ldap:725.v3cb_b_711b_1a_ef
locale:431.v3435fa_8f8445
lockable-resources:1255.vf48745da_35d0
mailer:472.vf7c289a_4b_420
matlab:2.13.1
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-449.v6960d7c54c69
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
multiple-scms:0.8
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pam-auth:1.10
parameterized-trigger:787.v665fcf2a_830b_
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-graph-view:242.v04d14822e97f
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
pipeline-stage-view:2.34
pipeline-utility-steps:2.16.2
plain-credentials:179.vc5cb_98f6db_38
plugin-util-api:4.1.0
prism-api:1.29.0-13
pubsub-light:1.18
resource-disposer:0.23
role-strategy:717.v6a_69a_fe98974
run-condition:1.7
scm-api:690.vfc8b_54395023
script-security:1335.vf07d9ce377a_e
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
sse-gateway:1.26
ssh-agent:367.vf9076cd4ee21
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.948.vb_8050d697fec
sshd:3.322.v159e91f6a_550
structs:337.v1b_04ea_4df7c8
terraform:1.0.10
theme-manager:215.vc1ff18d67920
thinBackup:2.0
timestamper:1.26
token-macro:400.v35420b_922dcb_
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
versioncolumn:243.vda_c20eea_a_8a_f
warnings-ng:11.3.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3894.vd0f0248b_a_fc4
workflow-durable-task-step:1336.v768003e07199
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:896.v175a_a_9c5b_78f
ws-cleanup:0.45
xunit:3.1.4

2

Hello @rvh, and welcome to this community. :wave:

The error message you’re seeing suggests there’s an issue with resuming the pipeline after it has been paused or interrupted.
This can happen if the build was paused and Jenkins was restarted before the build was resumed, or if there was an issue with the Jenkins controller or agent that caused the build to be interrupted.
The java.lang.IllegalStateException: Can't send a remote FilePath to a different remote channel error suggests there’s an issue with the file path of the build.
This could be due to several reasons, such as a network issue, a problem with the file system, or a configuration issue with Jenkins or the agent.

3

Hello @poddingue, thanks for the warm welcome and help. I’m not very experience with Jenkins yet,

The issue is that the build is not paused, it was actually fully completed already when this message shows up. When you refer to ‘Jenkins’ being restarted, do you mean the controller or the agent? To debug the issue I have created builds in the same project that would run very quickly, such that I could verify that during that build, the controller nor agent would restart.

Could it be that this situation happened once during a previous build, and now is still impacting newer builds on the same project somehow? I have had to ‘kill’ a build because it was preventing our controller from restarting. Could that somehow give issues now on newer builds? I saw in the documentation that this was not recommended, but could not find what the consequences could be in cases where you would do this.

Is there any direction I could take to further debug the FilePath issue? I’m not very familiar with the underlying mechanisms involved so I wouldn’t really know where to start looking.

either way, thank you for you reply!

1 Like
4

I have just been provided with credentials to remote into the system that hosts the jenkins controller. This provide me with some more information / patterns, that might be helpful in figuring out what is going on?

I have found that all affected builds contain folders named both “workflow-completed” as well as “workflow-backup”. These contain 2 <#>.xml files with just consecutive numbers as names (e.g., 50.xml and 51.xml), where the second file will contain the text to the flownode error.
The number in the filename always relates to the flownode that could not be loaded, so 51.xml will contain:
Cannot resume build because FlowNode 48 for flowHead 1 could not be loaded
while 27.xml will contain:
Cannot resume build because FlowNode 24 for flowHead 1 could not be loaded, can be found in 27.xml.

Aside from this, I also found what seemed to be an incomplete .xml file in the folder related to our develop branch with contents:

<?xml version='1.1' encoding='UTF-8'?>
<flow-definition plugin="workflow-job@1360.vc6700e3136f5">
  <actions>
    <org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobAction plugin="pipeline-model-definition@2.2150.v4cfd8916915c"/>
    <org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobPropertyTrackerAction plugin="pipeline-model-definition@2.2150.v4cfd8916915c">
      <jobProperties/>
      <triggers/>
      <parameters/>
      <options/>
    </org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobPropertyTrackerAction>
  </actions>
  <displayName>develop</displayName>
  <keepDependencies>false</keepDependencies>
  <properties>
    <org.jenkinsci.plugins.workflow.multibranch.BranchJobProperty plugin="workflow-multibranch@756.v891d88f2cd46">
      <branch plugin="branch-api@2.1128.v717130d4f816">
        <sourceId>db0fdd14-c5f7-4541-9adb-b57a7b4369eb</sourceId>
        <head class="jenkins.plugins.git.GitBranchSCMHead" plugin="git@5.2.1">
          <name>develop</name>
        </head>
        <

I have now removed this incomplete file in the hopes that this was maybe causing some issues, but unfortunately the issue persists. I’m also not entirely sure on how to proceed in order to repair this.

5

It appears as if it was an issue with the Matlab plugin. I have documented the issue on the github page of the plugin should anyone encounter similar issues:

use of runMATLABBuild in jenkinsfile, breaks builds · Issue #320 · mathworks/jenkins-matlab-plugin (github.com)

1 Like
6

Thanks for the feedback. :+1: