Variable scoping recently broke or changed?

Using Jenkins
1

Hi All,

I’m hoping someone can shed some light on an issue we’re having with our Jenkins jobs… Since ~Nov 24 it seems our jobs *not pipeline jobs) which declare and assign a variable in one section are now unable to reference the valie of this variable in another section - the retuned valie is now empty, it’s as if the scoping has changed to the local-to-the-section whereas before it was global-to-the-script.

As such, all of our builds and deployments are now failing. I’m at a loss to explain why this has changed or how to fix it. Our server is patched ny IT once a month, maybe it was caused by an update?

For e.g.

Execute Shell/Command:
rpmbuild --define "_topdir ${WORKSPACE}/rpmbuild" --define="release ${BUILD_NUMBER}" -ba rpmbuild/SPECS/vcre

$rpmbuild used to be accessable to all sections in the job, but now it evaluates to “”

→
Jenkins setup:
Jenkins: 2.401.1
OS: Linux - 4.18.0-553.30.1.el8_10.x86_64
Java: 11.0.25 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

active-directory:2.30
ant:481.v7b_09e538fcca
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
authentication-tokens:1.4
blueocean-commons:1.27.1
blueocean-core-js:1.27.1
blueocean-jwt:1.27.1
blueocean-pipeline-scm-api:1.27.1
blueocean-rest:1.27.1
blueocean-web:1.27.1
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
build-monitor-plugin:1.13+build.202205140447
build-timeout:1.28
build-user-vars-plugin:1.9
caffeine-api:3.1.8-133.v17b_1ff2e0599
cloud-stats:267.v577e3742c282
cloudbees-folder:6.800.v71307ca_b_986b
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.17.0-84.vb_b_938040b_078
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
deployit-plugin:23.1.0
disk-usage:0.28
display-url-api:2.204.vf6fddd8a_8b_e9
docker-commons:1.21
docker-java-api:3.2.13-68.va_875df25a_b_45
docker-plugin:1.3.0
docker-workflow:563.vd5d2e5c4007f
durable-task:504.vb10d1ae5ba2f
envinject-api:1.199.v3ce31253ed13
git:5.0.0
git-client:4.1.0
github:1.37.0
github-api:1.303-400.v35c2d8258028
github-branch-source:1701.v00cc8184df93
gradle:2.3
groovy:453.vcdb_a_c5c99890
icon-shim:3.0.0
instance-identity:142.v04572ca_5b_265
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.1
jersey2-api:2.38-1
jjwt-api:0.11.5-77.v646c772fddb_0
job-import-plugin:3.6
ldap:682.v7b_544c9d1512
leastload:3.0.0
mailer:448.v5b_97805e3767
matrix-auth:3.1.6
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
next-build-number:1.8
okhttp-api:4.10.0-125.v3593b_a_f8c97b_
pam-auth:1.10
pipeline-build-step:2.18.1
pipeline-github-lib:38.v445716ea_edda_
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.31
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.31
plain-credentials:143.v1b_df8b_d3b_e48
popper2-api:2.11.6-2
pubsub-light:1.17
python:1.3
resource-disposer:0.21
rpmsign-plugin:0.5.1
scm-api:676.v886669a_199a_a_
script-security:1369.v9b_98a_4e95b_2d
simple-theme-plugin:146.v0e67db_a_9052e
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
timestamper:1.22
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1010.vf7a_b_98e847c1
workflow-cps:3618.v13db_a_21f0fcf
workflow-durable-task-step:1234.v019404b_3832a
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:733.v109046189126
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:920.v59f71ce16f04
yet-another-docker-plugin:0.2.0

2

You say they are NOT pipeline jobs? How are you defining the rpmbuild variable?

3

Maybe it’s a pipeline jobm I’m not 100% sure as I didn’t create them. The variable is defined inside a Shell/Execute command section as:

rpmbuild --define "_topdir ${WORKSPACE}/rpmbuild" --define="release ${BUILD_NUMBER}" -ba rpmbuild/SPECS/vcre

4

I think it’s a freestlr/general job, the Configure entries contain Build Step etc but no pipeline section

5

This should have never worked, each Shell/Execute Command is a separate invocation of the shell, which means the scoping does not carry over from one command section to the next.

6

Thanks, I figured it would be down to scoping somehow. I’m not sure how it used to manage to work…

Is there a way I can share variables between the shells. Can I simply dot source in a properties file.?

7

The variable values are currently written to properties.txtI’m not sure if that’s just for logging purposes or whether jenkins shells may inherit env vars from the txt file, perhaps?

8

I don’t see anywhere that you use $rpmbuild or an env variable with name rpmbuild
It would help if you provide a screenshot of the job config (redact sensitive information) and the log of a failing job
rpmbuild is btw a command that you have on certain linux distributions (e.g. readhat and suse) that allows to build rpm packages (installer packages). The package that provides this command might have been removed by your IT.

9

OK thanks, here’s a fresh example with some screenshots and logfiles and I’ve annotated the steps…

First, we have a build step which sets version and rpmtype into properties.txt. I don’t know if properties.txt has any special meaning in Jenkins or whether we’re just lpgging here…

properties
properties1449×1146 49.7 KB

Next we run rpmbuild in a separate command/execute shell block. I can’t add a screenshot because new users are limited on one screenshot, but here is the text from the command/exec shell:

rpmbuild --define “_topdir {WORKSPACE}/rpmbuild" --define="release {BUILD_NUMBER}” -ba rpmbuild/SPECS/${rpmtype}.spec

The issue is that ${rpmtype}. used to expand to ‘vcsync’ but now it’s an empty string. If it were just this one job, I’d say someone must have deleted some code that dot sourced properties.txt, but we have about 15 jobs that are all exhibiting the same behaviour. Maybe there’s a plugin that automativally inherits from properties.txt and it stopped wotking after a patch or something?

Here’s the relevant part of the console output from the job:

–log snip–

  • cat properties.txt
    version = develop
    rpmtype = vcsync
    [vcsync-develop] $ /bin/sh -xe /tmp/jenkins4703385654598334318.sh
  • rpmbuild --define ‘_topdir /var/lib/jenkins/workspace/vcsync-develop/rpmbuild’ ‘–define=release 36’ -ba rpmbuild/SPECS/.spec
    error: failed to stat /var/lib/jenkins/workspace/vcsync-develop/rpmbuild/SPECS/.spec: No such file or directory
    –eonf of log snip—

The build fails because the rpmbuild command has been constructed with ‘/SPECS/.spec’ instead of ‘/SPECS/vcsync.spec’

TIA
Duncan

10

It is possible to inject env variables between steps in freestyle jobs. But it requires an explicit setp from the Environment Injector plugin
With the step Inject environment variables you can give a path to a properties file.
But without this there is no way that a properties.txt is implicitly loaded

11

Thanks for the info Markus. We must have had something like Environment Injector working not so long ago.I’ll look into it.