JENKINS-48300: wrapper script does not seem to be touching the log file

Using Jenkins Ask a question
1

Hey Team,
We have noticed one of our Jenkins Pipeline fail with

wrapper script does not seem to be touching the log file in /home/jenkins/agent/workspace/disqa-1111-jenkins-pipeline-test@tmp/durable-3bdc99b0
(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)

after running for about ~3h. :question: Why and when do we see this issue?

1). We don’t wish to

    script {       System.setProperty("org.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL", "86400");
    }

since setting this within the Jenkinsfile of the pipeline will make the changes at the Admin-level. We are looking for options restricted only that PipelineJob.

2). Tried checking the log file generated at disqa-1111-jenkins-pipeline-test@tmp/durable-3bdc99b0
– The root user creates a log file jenkins-log.txt within directory durable-3bdc99b0/. We are using kubernetes cloud config to run the pipeline job, and all the other files are managed by user podman.

[root@ocas-build-784c01bd-8bd6-4464-9569-28169d63ba1d-qw9h6-88djm durable-b11cc10b]# ls -lart
total 8
drwxrwxrwx. 5 podman podman   73 Jul  8 19:11 ..
-rw-r--r--. 1 podman podman  107 Jul  8 19:11 script.sh
drwxr-xr-x. 2 podman podman   46 Jul  8 19:11 .
-rw-r--r--. 1 root   root   3820 Jul  8 19:14 jenkins-log.txt

Is the user permissions resulting in the issue above?

– I tried to manually create the jenkins-log.txt with correct user & file permissions, but not sure, the durable-* folder has a different sequence while performing linux manipulations in the Jenkinsfile

                                    def durableDir = sh(script: "ls `pwd`@tmp | grep -i 'durable'", returnStdout: true).trim()
                                    def logDir = "`pwd`@tmp/${durableDir}"
                                    sh """
                                        mkdir -p $logDir
                                        touch $logDir/jenkins-log.txt
                                        chown -R podman:podman $logDir/jenkins-log.txt
                                        chmod 777 $logDir/jenkins-log.txt
                                        chown -R podman:podman $logDir
                                    """

When I checked the path within the running pod;

[root@ocas-build-784c01bd-8bd6-4464-9569-28169d63ba1d-qw9h6-88djm disqa-1111-jenkins-pipeline-test@tmp]# ls -la
total 0
drwxrwxrwx. 5 podman podman  73 Jul  8 19:11 .
drwxr-xr-x. 4 podman podman 112 Jul  8 19:07 ..
drwxr-xr-x. 2 podman podman  29 Jul  8 19:11 durable-89909675
drwxr-xr-x. 2 podman podman  46 Jul  8 19:11 durable-b11cc10b
drwx------. 2 podman podman   6 Jul  8 19:11 secretFiles

The durable-* folders are different.

3). Will including

sh "export JAVA_OPTS=Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true"

help resolve the issue?

Please let me know if there are any other suggestions/solutions to fix this issue.
Regards
Hema

2

From my experience using the binary wrapper is the best solution. We also had this error from time to time (about 5 to 10 occurrences per day at 2000 builds per day). Usually this happened when the build agents were under heavy load or memory pressure. Root cause was not clear but somehow the wrapper script itself crashed or was killed.
After switching to the binary wrapper the problem disappeared completely.
Note that you must enable the binary wrapper by setting the java property on the controller vm at startup (to enable it permanently) or via the script console (just for the running Jenkins, lost on restart) , your approach in 3) will not work.

Your idea in 2) can’t work because jenkins will create for each sh step a temporary directory with the mentioned files and deletes them when the sh step finished.

1 Like
3

Will set Dorg.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true via the script-console to check if it works for us!

:question: Any functional difference between shell and binary wrapper?

4

it should be org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true

No functional difference between the two

1 Like
5

System.setProperty("org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER", "true");

Setting this property still errored with

wrapper script does not seem to be touching the log file in /home/jenkins/agent/workspace/disqa-1111-jenkins-pipeline-test@tmp/durable-edf21dc6
(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)

Is there anything which can be tried further?

6

if you want to change this for a running instance you will need to run this in the script console
org.jenkinsci.plugins.durabletask.BourneShellScript.USE_BINARY_WRAPPER=true

Using System.setProperty will not work as the property is read only once at startup of Jenkins

1 Like