JENKINS-48300 extremely laggy filesystem

Using Jenkins Ask a question
1

Hello,
During the execution of the jenkins declartivie pipeline; one of the stages step shell script throws below generic error.
I have tried to research about this error code over the internet. Not able to find a solution. Below solution seems to be promising but it didn’t worked for me[1].
Here are the steps:

  1. Our current Jenkins Controller and Jenkins Agent Architecture is serverless[2]
  2. Jenkins Agents build docker images using Kaniko[3]
  3. It is successfully able to build the docker image and push to the ECR. Shell Script doesn’t exit and go to the next stage as below screenshot.
    image
    image1440×552 15.5 KB

    I appreciate your responses.
    References:
    1.[JENKINS-50379] Jenkins kills long running sh script with no output - Jenkins Jira
2
  1. Building a serverless Jenkins environment on AWS Fargate | AWS DevOps Blog
  2. Serverless Jenkins on Fargate with Kaniko Agent - Shine Solutions Group %
3

I notice there is an error that the command “touch” is not found. Is your path changed such that touch wouldn’t be found in the path?

5

I don’t think there is any issue with the path location.

image
image751×72 1.81 KB

6

Here is the actual step.

image
image843×552 8.03 KB

7

Please anyone has a solution or workaround for this issue to be mitigated.

8

There’s a lot going on in that screenshot. What is your question? How to work around a bug? I’m sure the ticket has more details than we do. Why touch isn’t found? It’s either not installed, not in path or wrong arch

9

@halkeye My question is why the temporary durable shell task step hangs after successful execution of last multiline shell script command. The bash command touch not found seems to be misguiding me. Under the hood if there is something happening i don’t have any clue. I am trying to understand if anyone experienced similar issue with kaniko container image build or any known workaround solutions.

10

Looks like there’s a background script that runs touch on the log file every 3 seconds

Then the error message is triggered when the log hasn’t been updated in 2-3 seconds.

Considering your screenshot says touch isn’t found, its likely the culpret. I don’t know why you are getting the response of touch not found. Maybe the shebang, does /busybox/sh have a working touch? if you run touch /dev/null does it error? how about arch $(which touch) or ldd $(which touch) which may tell you if you are using the wrong arch or missing libraries or something.

11

@halkeye Thanks for the above details. Few observations to your points. It doesn’t matter what shebang i use it #!/bin/bash or #!/busybox/sh the behaviour is consistent. touch command is not found after the shell command kaniko executor. Below is the screenshots of the output.

image
image1508×359 20.5 KB

12

I wonder if kaniko messes with path or anything. Maybe try an echo $PATH maybe echo $SHELL won’t hurt either.

It very much doesn’t feel like a jenkins issue which is making it harder and harder to debug. I’ve also never used kaniko so i don’t know how it works. It feels like its messing with your shell somehow.

13

@halkeye i found the root cause of the issue. It is due to Kaniko executor argument --skip-unused-stages=true. After removing it. All the things are working as expected.

14

see: Kaniko Container can't exectute sh command · Issue #1212 · GoogleContainerTools/kaniko · GitHub

stages {
        stage('Main') {
            environment {
                PATH = "/busybox:/kaniko:$PATH"
            }
            steps {
                container(name: 'kaniko', shell: '/busybox/sh') {
                    sh '''#!/busybox/sh
                    echo "in kaniko"
                    '''
                }
            }
        }
    }