Junit processing hangs and eventually kills remoting agent

Using Jenkins Ask a question
1
Jenkins setup: 
Jenkins: 2.504.2
OS: Linux - 6.1.127-135.201.amzn2023.x86_64
Java: 21.0.7 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
analysis-model-api:13.2.0
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.5-150.veb_76e719855b_
artifact-manager-s3:926.ve7c0b_ffccd20
asm-api:9.8-135.vb_2239d08ee90
authentication-tokens:1.131.v7199556c3004
aws-credentials:248.v78a_dcfc9db_ff
aws-global-configuration:146.vfcec61593eea_
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.31.45-50.vee50cf9f54c3
aws-java-sdk2-ec2:2.31.45-50.vee50cf9f54c3
aws-java-sdk2-s3:2.31.45-50.vee50cf9f54c3
aws-secrets-manager-secret-source:1.72.v61781b_35c542
badge:2.8
basic-branch-build-strategies:228.v68c089762a_db_
blueocean:1.27.20
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.20
blueocean-commons:1.27.20
blueocean-config:1.27.20
blueocean-core-js:1.27.20
blueocean-dashboard:1.27.20
blueocean-display-url:2.4.4
blueocean-events:1.27.20
blueocean-executor-info:1.27.20
blueocean-git-pipeline:1.27.20
blueocean-github-pipeline:1.27.20
blueocean-i18n:1.27.20
blueocean-jira:1.27.20
blueocean-jwt:1.27.20
blueocean-personalization:1.27.20
blueocean-pipeline-api-impl:1.27.20
blueocean-pipeline-editor:1.27.20
blueocean-pipeline-scm-api:1.27.20
blueocean-rest:1.27.20
blueocean-rest-impl:1.27.20
blueocean-web:1.27.20
bootstrap5-api:5.3.5-1
bouncycastle-api:2.30.1.80-261.v00c0e2618ec3
branch-api:2.1217.v43d8b_b_d8b_2c7
build-monitor-plugin:1.14-985.v7b_f37b_3d0b_f5
build-pipeline-plugin:2.0.2
build-timeout:1.38
build-timestamp:1.1.0
caffeine-api:3.2.0-166.v72a_6d74b_870f
checks-api:370.vb_61a_c57328f3
cloudbees-bitbucket-branch-source:936.4.0
cloudbees-folder:6.1023.v4fcb_72152519
command-launcher:123.v37cfdc92ef67
commons-collections4-api:4.5.0-8.va_d5448ef9011
commons-compress-api:1.27.1-3
commons-httpclient3-api:3.1-3
commons-lang3-api:3.17.0-87.v5cf526e63b_8b_
commons-text-api:1.13.1-176.v74d88f22034b_
compress-buildlog:1.2
conditional-buildstep:1.5.0
configuration-as-code:1971.vf9280461ea_89
copyartifact:770.va_6c69e063442
coverage:1.16.1
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
dashboard-view:2.528.v3470c02b_d7c9
data-tables-api:2.2.2-1
depgraph-view:1.0.5
display-url-api:2.209.v582ed814ff2f
docker-commons:457.v0f62a_94f11a_3
docker-workflow:621.va_73f881d9232
downstream-build-cache:1.7
dtkit-api:3.0.3
durable-task:587.v84b_877235b_45
ec2:1935.v9b_77210472b_1
echarts-api:5.6.0-4
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
embeddable-build-status:548.v5653c6e28c41
emoji-symbols-api:16.0-26.v9818ff7423f0
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
external-monitor-job:223.vb_fddcf42c9b_3
favorite:2.237.v79163ca_8b_892
font-awesome-api:6.7.2-1
forensics-api:3.1.0
git:5.7.0
git-client:6.1.3
git-parameter:439.vb_0e46ca_14534
git-server:137.ve0060b_432302
github:1.43.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1822.v9eec8e5e69e3
github-checks:602.v264a_83610da_6
github-scm-trait-notification-context:45.v8ef831829589
global-slack-notifier:1.5
golang:1.4
gradle:2.15
gson-api:2.13.1-139.v4569c2ef303f
handy-uri-templates-2-api:2.1.8-36.v85e4cb_234a_13
htmlpublisher:425
http_request:1.20
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:88.va_4187cb_eddf1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jenkins-design-language:1.27.20
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.17
jjwt-api:0.11.5-120.v0268cf544b_89
jobConfigHistory:1328.vb_24fa_565cfec
joda-time-api:2.14.0-127.v7d9da_295a_d51
jquery:1.12.4-3
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250517-153.vc8a_a_d87c0ce3
json-path-api:2.9.0-148.v22a_7ffe323ce
jsoup:1.20.1-46.ve5f1416988c2
junit:1335.v6b_a_a_e18534e1
junit-attachments:299.v2e00f511b_538
ldap:780.vcb_33c9a_e4332
lockable-resources:1349.v8b_ccb_c5487f7
log-parser:2.5.0
mailer:489.vd4b_25144138f
matrix-auth:3.2.6
matrix-project:849.v0cd64ed7e531
maven-plugin:3.26
mercurial:1309.v6802b_f0efb_b_9
mina-sshd-api-common:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-core:2.15.0-161.vb_200831a_c15b_
mina-sshd-api-scp:2.15.0-161.vb_200831a_c15b_
monitoring:2.5.0
nexus-artifact-uploader:2.14
node-iterator-api:72.vc90e81737df1
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:324.v432cce4172ca_
pam-auth:1.12
parameterized-trigger:859.vb_e3907a_07a_16
pipeline-aws:1.45
pipeline-build-step:567.vea_ce550ece97
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:241.vc3d48fb_b_2582
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:527.vd61b_1d3c5078
pipeline-milestone-step:138.v78ca_76831a_43
pipeline-model-api:2.2255.v56a_15e805f12
pipeline-model-definition:2.2255.v56a_15e805f12
pipeline-model-extensions:2.2255.v56a_15e805f12
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2255.v56a_15e805f12
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:195.vb_906e9073dee
plot:2.4.0
plugin-util-api:6.1.0
prism-api:1.30.0-1
pubsub-light:1.19
pyenv-pipeline:2.1.2
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
run-condition:243.v3c3f94e46a_8b_
scm-api:704.v3ce5c542825a_
script-security:1373.vb_b_4a_a_c26fa_00
simple-theme-plugin:211.v5424a_5510e47
skip-certificate-check:50.vcfa_cdb_b_27312
slack:761.v2a_8770f0d169
snakeyaml-api:2.3-125.v4d77857a_b_402
sse-gateway:1.28
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:350.v3b_30f09f2363
test-results-analyzer:309.vda_3a_a_f100542
timestamper:1.29
token-macro:444.v52de7e9c573d
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
view-job-filters:401.va_809f6a_b_0c26
vncrecorder:1.35
vncviewer:1.9
warnings-ng:12.6.0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1373.v7b_813f10efa_b_
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4106.v7a_8a_8176d450
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1532.va_9a_d244074a_3
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48
xunit:3.1.5
xvnc:1.28
yet-another-build-visualizer:1.17

Controller Amazon linux 2003 and agents are Rocky Linux 9.6.

Reproduction steps

  1. rsync previous creation of XML results to a new agent node. (these are normally produced as part of a job)
  2. copy these files into the job workspace
  3. run junit on those files (with no additional arguments)

Jenkins pipeline section contains simply:

stage('JUnit Analysis') {
  junit testResults: "file.xml", keepLongStdio: false
}

Expected Results

Junit processes the XML files within

Actual Results

junit hangs until it times out

[Pipeline] junit
Recording test results
Cancelling nested steps due to timeout
null

The agent connection dies as if completely hung although I can still ssh onto it and can see it’s not loaded. We provision a single agent per job so there’s nothing else happening on the node.

Anything else?

The XML file is 32 Mb in size. If I split it into smaller chunks of XML and loop over each one (500k-1.5mb) it will process 23-26 files, even if I change the order or remove files that were previously stuck, before hanging again.

Note that this is running on a new Jenkins environment and the XML processing is working on an older installation.

Thread dump of the running agent during the hang thread.txt

I wondered if it is some kind of resource exhaustion on the agent? I tried increasing memory (-Xmx) in the agent launch command but it didn’t help.

Does anyone have any ideas?

Note: this is a copy of an issue I created here that hasn’t seen much love hence tring here.

2

I can’t duplicate the issue that you’re seeing with the data that you’ve provided in the GitHub issue.

Here is the Pipeline job that I ran:

pipeline {
    agent {
        label '!windows'
    }
    stages {
        stage('Download-test-results') {
            steps {
                sh '[ -f aws-rocky-810-x64-GmZkSP-pm-default-0.xml.gz ] || wget https://github.com/user-attachments/files/19871462/aws-rocky-810-x64-GmZkSP-pm-default-0.xml.gz'
            }
        }
        stage('Unzip-test-results') {
            steps {
                sh '[ -f aws-rocky-810-x64-GmZkSP-pm-default-0.xml ] || gunzip --keep aws-rocky-810-x64-GmZkSP-pm-default-0.xml.gz; ls'
            }
        }
        stage('JUnit') {
            steps {
                junit testResults: 'aws-rocky-810-x64-GmZkSP-pm-default-0.xml'
            }
        }
    }
}
3

Hi Mike,

thanks for your reply and giving it a try. I think that it might be a remoting issue. What kind of agent connection method do you use? We have EC2 Plugin launching AWS instances and using the remoting.jar via SSH. I tried using agent.jar and websocket yesterday which seemed more stable.

Thanks,

Matt

4

Issues that are ultimately the fault of Jenkins remoting are very, very rare.

I have static agents that use ssh to local computers, static agents that use ssh to cloud hosted computers, static agents on local computers that use an inbound connection, and ephemeral agents that use an inbound connection.

5

If I create a static agent as an inbound connection, launch an AWS instance and use websocket method with agent.jar, I can run my simple test pipeline multiple times and it seems to parse the data in a few seconds. If I use the EC2 plugin which uses remoting.jar and I presume the TCP connection method it hangs on the first or second run. I’ve tried updating the agent OS from Rocky 8 to 9, JDK from 17 to 21. And with Ubuntu 24.04 as an agent. All the same behaviour.

Do you have any ideas what I could do next or check?

The file I provided is just a sample of the data. The issue occurs when the processing 32mb of data. If I split the data into smaller files it is less likely but still happens.

I’m starting to think I might need to try another way to provisioning ephemeral instances with inbound connections.

Many thanks for your input.

6

Check the networking configuration between the problematic agent and the controller.

7

I found the issue. Wrote up the reason in the linked issue here

Basically, it was because the timestamp format wasn’t in an ISO format acceptable to the Java parser method. This was generating 1000s of exceptions/log events. This must have caused a resource issue and hung the agent process.

1 Like