Built-In Node Not Working After Plugin Upgrades

Using Jenkins Ask a question
1

Jenkins setup:
Jenkins: 2.504.1
OS: Linux - 4.14.193-113.317.amzn1.x86_64
Java: 17.0.15 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

ace-editor:1.1
ant:1.11
antisamy-markup-formatter:173.v680e3a_b_69ff3
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
apache-httpcomponents-client-5-api:5.4.4-144.vc483d91903d2
asm-api:9.8-135.vb_2239d08ee90
authentication-tokens:1.131.v7199556c3004
aws-credentials:245.v8a_1b_7c11a_94d
aws-global-configuration:146.vfcec61593eea_
aws-java-sdk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-api-gateway:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-autoscaling:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudformation:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudfront:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-cloudwatch:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codebuild:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-codedeploy:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ec2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecr:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ecs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-efs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticbeanstalk:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-elasticloadbalancingv2:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-iam:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-kinesis:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-lambda:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-logs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-minimal:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-organizations:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-secretsmanager:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sns:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-sqs:1.12.780-480.v4a_0819121a_9e
aws-java-sdk-ssm:1.12.780-480.v4a_0819121a_9e
aws-java-sdk2-core:2.31.26-37.vd178d9b_0558f
aws-java-sdk2-ec2:2.31.26-37.vd178d9b_0558f
azure-ad:561.v85263ce9a_b_b_f
azure-commons:1.0.4
azure-sdk:211.vd3b_6c5504331
bootstrap4-api:4.6.0-3
bootstrap5-api:5.3.5-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1217.v43d8b_b_d8b_2c7
build-name-setter:2.5.0
build-timeout:1.38
caffeine-api:3.2.0-161.v691ef352cee1
checks-api:370.vb_61a_c57328f3
cloudbees-folder:6.1012.v79a_86a_1ea_c1f
command-launcher:123.v37cfdc92ef67
commons-compress-api:1.27.1-3
commons-lang3-api:3.17.0-87.v5cf526e63b_8b_
commons-text-api:1.13.0-153.v91dcd89e2a_22
config-file-provider:988.v0461fcc2b_9d1
copyartifact:770.va_6c69e063442
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
data-tables-api:2.2.2-1
display-url-api:2.209.v582ed814ff2f
docker-commons:451.vd12c371eeeb_3
docker-workflow:611.v16e84da_6d3ff
durable-task:587.v84b_877235b_45
echarts-api:5.6.0-4
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
font-awesome-api:6.7.2-1
git:5.7.0
git-client:6.1.3
git-parameter:439.vb_0e46ca_14534
git-server:137.ve0060b_432302
github:1.43.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1815.v9152b_2ff7a_1b_
gradle:1.36
gson-api:2.12.1-113.v347686d6729f
handlebars:1.1.1
instance-identity:203.v15e81a_1b_7a_38
ionicons-api:88.va_4187cb_eddf1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:327.vdfe586651ee0
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jjwt-api:0.11.5-120.v0268cf544b_89
joda-time-api:2.14.0-127.v7d9da_295a_d51
jquery:1.12.4-3
jquery-detached:1.2.1
jquery3-api:3.7.1-3
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
jsoup:1.20.1-46.ve5f1416988c2
junit:1334.vd3b_b_2094e438
ldap:780.vcb_33c9a_e4332
lockable-resources:1349.v8b_ccb_c5487f7
mailer:489.vd4b_25144138f
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.5
matrix-project:847.v88a_f90ff9f20
maven-plugin:3.26
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
momentjs:1.1.1
multiple-scms:0.8
nodejs:1.6.4
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:324.v432cce4172ca_
pam-auth:1.6
pipeline-build-step:567.vea_ce550ece97
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:237.v2b_75640ca_888
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-milestone-step:134.vdf60d179845f
pipeline-model-api:2.2255.v56a_15e805f12
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:2.2255.v56a_15e805f12
pipeline-model-extensions:2.2255.v56a_15e805f12
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2255.v56a_15e805f12
pipeline-stage-view:2.38
pipeline-utility-steps:2.19.0
plain-credentials:195.vb_906e9073dee
plugin-util-api:6.1.0
popper-api:1.16.1-2
popper2-api:2.11.2-1
rebuild:338.va_0a_b_50e29397
resource-disposer:0.25
s3:498.v4c32ea_de55c2
saml:4.525.v4f6a_7209447e
scm-api:704.v3ce5c542825a_
script-security:1373.vb_b_4a_a_c26fa_00
snakeyaml-api:2.3-123.v13484c65210a_
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:343.vdcf37b_a_c81d5
subversion:1287.vd2d507146906
terraform:1.0.10
timestamper:1.28
token-macro:444.v52de7e9c573d
trilead-api:2.192.vc50a_d147e369
uno-choice:2.8.8
variant:70.va_d9f17f859e0
workflow-aggregator:608.v67378e9d3db_1
workflow-api:1373.v7b_813f10efa_b_
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4106.v7a_8a_8176d450
workflow-cps-global-lib:615.vb_b_0664a_b_19f3
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1520.v56d65e3b_4566
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48

Our Jenkins runs in AWS ECS and hadn’t been updated in about 4-5 years. Ieverything was running fine until the ECS service deployed a new container with the latest version of Jenkins. After the unexpected upgrade our multi-branch pipeline jobs were having some issues, but the pipeline was at least running until it hit the Terraform init stage. Then one of my coworkers update like 50 plugins all at once and then the jobs started completely failing. At this point the task of getting Jenkins working again got handed off to me.

From what I can tell the issue is that the Built-In Node is failing. We do have an alert saying that we need to apply a migration for the Built-In Node label. My question is what exactly happens when I click the “Apply Migration” button?

Any help would be greatly appreciated.

Here is the console output from a failed job.

Started by user

git rev-parse --resolve-git-dir /var/jenkins_home/caches/git-56517cdc21adad3caf67b9ef14fdae8d/.git # timeout=10
Setting origin to https://git-codecommit.us-east-1.amazonaws.com/v1/repos/
git config remote.origin.url https://git-codecommit.us-east-1.amazonaws.com/v1/repos/ # timeout=10
Fetching origin…
Fetching upstream changes from origin
git --version # timeout=10
git --version # ‘git version 2.39.5’
git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
git fetch --tags --force --progress – origin +refs/heads/:refs/remotes/origin/ # timeout=10
hudson.plugins.git.GitException: Command “git fetch --tags --force --progress – origin +refs/heads/:refs/remotes/origin/” returned status code 128:
stdout:
stderr: fatal: unable to access ‘https://git-codecommit.us-east-1.amazonaws.com/v1/repos//’: The requested URL returned error: 403

at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2852)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2188)
at PluginClassLoader for git-client//org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:638)
at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:413)

Caused: java.io.IOException
at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:419)
at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.doRetrieve(AbstractGitSCMSource.java:332)
at PluginClassLoader for git//jenkins.plugins.git.AbstractGitSCMSource.retrieve(AbstractGitSCMSource.java:441)
at PluginClassLoader for scm-api//jenkins.scm.api.SCMSource.fetch(SCMSource.java:581)
at PluginClassLoader for workflow-multibranch//org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:104)
at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:315)
at hudson.model.ResourceController.execute(ResourceController.java:101)
at hudson.model.Executor.run(Executor.java:446)
Finished: FAILURE

2

The migration will change the label from master to built-in. So this will affect jobs that want to explicitly run on the controller node (e.g. by using node("master") ... in a scripted pipeline).
But this is not the problem you’re facing. There might be something wrong with the credentials that are used to checkout the Jenkinsfile as Jenkins gets a forbidden return code.

3

Thank you for explaining the migration. As you mentioned, the root issue appears to be that Jenkins cannot authenticate to the CodeCommit repository. I have tested the credentials, DNS, and the firewall. All appear to be configured properly. Our environment runs on an AWS autoscaling group Linux host with ECS Linux containers running Jenkins, so the deployment environment is very consistent and hasn’t changed. Everything worked until the ECS container updated Jenkins (task definition set to jenkins:lts) and then we updated all the plugins. Also, there is nothing in the CodeCommit logs indicating that Jenkins has attempted to authenticate. As far as I can tell Jenkins is not actually connecting to the CodeCommit access point. I believe that all of the infrastructure and network are configured correctly. So my assumption is that something inside Jenkins is not properly executing a call to CodeCommit. Because our Built-In node has a warning for 0B swap file I was hoping that it might be a node issue due to the label migration needing to be done.

Is it possible that the password or network connectivity got changed or corrupted within Jenkins during all the upgrades? If yes, how could I test, and where would I need to look? I have been chasing this problem for over a week and I am running out of ideas on where to look. Any help our guidence would be greatly appreciated.

4

Here is another bit of information. I ran the script below in the console and got this result.

Git connection failed. Exit code: 128
An error occurred: No such property: errorText for class: java.lang.ProcessImpl

This appears to be the same exit code that I am seeing when my job failes to start. But I don’t know what the java.lang.ProcessImpl is.

def gitUrl = ‘https://git-codecommit.us-east-1.amazonaws.com/v1/repos/’ // Replace with your Git URL

try {
def process = “git ls-remote $gitUrl”.execute()
process.waitFor()

if (process.exitValue() == 0) {
    println "Git connection successful!"
    println "Output:\n${process.text}"
} else {
    println "Git connection failed. Exit code: ${process.exitValue()}"
    println "Error output:\n${process.errorText}"
}

} catch (Exception e) {
println “An error occurred: ${e.getMessage()}”
}

5

Well it is obvious that you need authentication when you connect to your git server. So it is no surprise that git ls-remote failed. Do you have this problem with all jobs and credentials that you use?
Maybe for some reason the credentials store was corrupted, then the passwords would be wrong probably. Not sure if a corruption would lead to any errors during startup though.
You could check with a simple pipeline where you extract the password and echo it into a file it is still correct.

6

ProcessImpl is a class from Java that starts another process.