Rebuilding Jenkins Debian package locally with the `packaging` repository

Hi there :wave: ,
I’m trying to rebuilt locally .deb packages because I’m having an issue with my slow armhf machines.

I started a brand new Ubuntu 22.04 LTS x86_64 virtual machine, forked (just in case) and cloned locally the packaging repo.
I then launched the ./prep.sh script, and then make deb.
I have created a gpg key with gpg --gen-key beforehand (following this howto), exported the key-id in the shell environment variables GPGKEY and GPG_KEYNAME, but I still get this error:

Now signing changes and any dsc files...
 signfile buildinfo jenkinstest_2.350_all.buildinfo Bogus user <bogus@example.org>
gpg: skipped "Bogus user <bogus@example.org>": No secret key
gpg: /tmp/debsign.kueYa4bD/jenkinstest_2.350_all.buildinfo: clear-sign failed: No secret key
debsign: gpg error occurred!  Aborting....
debuild: fatal error at line 1112:
running debsign failed
+ rm -rf /tmp/tmp.pRvVVOPN6S
make: *** [Makefile:40: target/debian/jenkinstest_2.350_all.deb] Error 29

As you may have already guessed :rofl: I’m not that familiar with gpg, so the error must be on my side.
Could someone please point out what my obvious mistake/oversight is? :pray:

Thanks.

After a few mistakes on my side (`createrepo` doesn't exist in Ubuntu `20.04` · Issue #314 · jenkinsci/packaging · GitHub), Current documentation does not work for me despite using Docker · Issue #316 · jenkinsci/packaging · GitHub, I got more information from core maintainers.
I’m not trying anymore to use a Ubuntu virtual machine, but directly the docker image supplied by the infra team.
I’m still using Windows unfortunately. I know that’s all but a recommended platform, but one got to start somewhere… And as I’m using docker I thought this would work nonetheless.
1. End of lines
The first issue I’m having is that despite adding a .gitattributes file and checking out in a new directory, well, I still have bad end of lines within docker (btw, thanks @dduportal for suggesting that).
A quick and dirty find . -name "*sh" -exec dos2unix {} \; -print under cygwin fixed the ./prep.sh : invalid option. Better do that with *.py files too, because you will face a /usr/bin/env: ‘python3\r’: No such file or directory later on.

I will now have to find within GitHub Desktop if I can change an option so that does not happen anymore. I also have the issue when using git clone on the command line, or gh repo clone gounthar/packaging.
I totally understand this is not an error linked in any way with Jenkins, but with the use of git under Windows. I also know Windows is not a recommended platform for developing with Jenkins, but I think going through docker should work.

2. make error
Once the end of lines is solved, I’m still having issues with the packaging under docker.
In the documentation, it is written:

Run ./prep.sh to perform the preparatory actions of downloading the WAR and importing the GPG key.

The script seems to work despite displaying an error:

 ./prep.sh
+ set -o pipefail
++ dirname ./prep.sh
+ cd .
+ [[ ! -f /srv/releases/jenkins/jenkins.war ]]
+ jv download
INFO[0000] Downloading version 2.354 from https://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/2.354/jenkins-war-2.354.war
INFO[0012] War downloaded to /srv/releases/jenkins/jenkins.war
INFO[0012] downloaded 86.6 MiB
+ gpg --fingerprint 'Bogus Test'
gpg: directory '/home/jenkins/.gnupg' created
gpg: keybox '/home/jenkins/.gnupg/pubring.kbx' created
gpg: /home/jenkins/.gnupg/trustdb.gpg: trustdb created
gpg: error reading key: No public key
+ gpg --import --batch /srv/releases/jenkins/credentials/sandbox.gpg
gpg: key 42715B56E211B042: public key "Bogus Test (This is test only key) <noreply@jenkins-ci.org>" imported
gpg: key 42715B56E211B042: "Bogus Test (This is test only key) <noreply@jenkins-ci.org>" not changed
gpg: key 42715B56E211B042: secret key imported
gpg: Total number processed: 2
gpg:               imported: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
+ exit 0

Next step is to invoke make.
In the documentation, I can read:

Run make package to build all the native packages. At minimum, you have to specify the WAR variable that points to the war file to be packaged and a branding file (for licensing and package descriptions). You will probably need to pass in the build environment and credentials.
For example:
make package BRAND=./branding/jenkins.mk BUILDENV=./env/test.mk CREDENTIAL=./credentials/test.mk

It looks like the WAR environment variable is set:

set | grep WAR
WAR=/srv/releases/jenkins/jenkins.war

I still get an error for the make command.

/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 2: usr/share/jenkins: not found`
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 5: var/lib/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 9: var/log/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 12: var/cache/jenkins: not found
dh_installdirs: warning: debian/jenkins.dirs is marked executable but does not appear to an executable config.
dh_installdirs: warning:
dh_installdirs: warning: If debian/jenkins.dirs is intended to be an executable config file, please ensure it can
dh_installdirs: warning: be run as a stand-alone script/program (e.g. "./debian/jenkins.dirs")
dh_installdirs: warning: Otherwise, please remove the executable bit from the file (e.g. chmod -x "debian/jenkins.dirs")
dh_installdirs: warning:
dh_installdirs: warning: Please see "Executable debhelper config files" in debhelper(7) for more information.
dh_installdirs: warning:
dh_installdirs: error: debian/jenkins.dirs (executable config) returned exit code 127
make: *** [debian/rules:4: binary-indep] Error 25

Here is the detail:

Detailed log
./deb/build/build.sh
+ hostname
a12c4b996b36
++ dirname ./deb/build/build.sh
+ dir=./deb/build
++ mktemp -d
+ D=/tmp/tmp.GaXJspqUF9
+ trap 'rm -rf "${D}"' EXIT
+ cp -R ./deb/build/build.sh ./deb/build/debian /tmp/tmp.GaXJspqUF9
+ cp /srv/releases/jenkins/systemd/jenkins.service /tmp/tmp.GaXJspqUF9/debian
+ cp /srv/releases/jenkins/systemd/jenkins.sh /tmp/tmp.GaXJspqUF9
+ cp /srv/releases/jenkins/systemd/migrate.sh /tmp/tmp.GaXJspqUF9
+ sed -i.bak -e 's/^\s*$/./' -e 's/^/ /' /srv/releases/jenkins/branding/description-file
+ /srv/releases/jenkins/bin/branding.py /tmp/tmp.GaXJspqUF9
+ mv /srv/releases/jenkins/branding/description-file.bak /srv/releases/jenkins/branding/description-file
+ cat
++ date -R
+ cp /srv/releases/jenkins/jenkins.war /tmp/tmp.GaXJspqUF9/jenkins.war
+ pushd /tmp/tmp.GaXJspqUF9
/tmp/tmp.GaXJspqUF9 /srv/releases/jenkins
+ pushd debian
/tmp/tmp.GaXJspqUF9/debian /tmp/tmp.GaXJspqUF9 /srv/releases/jenkins
+ for f in jenkins.*
+ mv jenkins.default jenkins.default_
++ echo jenkins.default
++ cut -b8-
+ mv jenkins.default_ jenkins.default
+ for f in jenkins.*
+ mv jenkins.dirs jenkins.dirs_
++ echo jenkins.dirs
++ cut -b8-
+ mv jenkins.dirs_ jenkins.dirs
+ for f in jenkins.*
+ mv jenkins.init jenkins.init_
++ echo jenkins.init
++ cut -b8-
+ mv jenkins.init_ jenkins.init
+ for f in jenkins.*
+ mv jenkins.install jenkins.install_
++ echo jenkins.install
++ cut -b8-
+ mv jenkins.install_ jenkins.install
+ for f in jenkins.*
+ mv jenkins.logrotate jenkins.logrotate_
++ echo jenkins.logrotate
++ cut -b8-
+ mv jenkins.logrotate_ jenkins.logrotate
+ for f in jenkins.*
+ mv jenkins.postinst jenkins.postinst_
++ echo jenkins.postinst
++ cut -b8-
+ mv jenkins.postinst_ jenkins.postinst
+ for f in jenkins.*
+ mv jenkins.postrm jenkins.postrm_
++ echo jenkins.postrm
++ cut -b8-
+ mv jenkins.postrm_ jenkins.postrm
+ for f in jenkins.*
+ mv jenkins.service jenkins.service_
++ echo jenkins.service
++ cut -b8-
+ mv jenkins.service_ jenkins.service
+ popd
/tmp/tmp.GaXJspqUF9 /srv/releases/jenkins
+ mv jenkins.sh jenkins
+ mv migrate.sh migrate
+ debuild -Zgzip -A
 dpkg-buildpackage -us -uc -ui -Zgzip -A
dpkg-buildpackage: info: source package jenkins
dpkg-buildpackage: info: source version 2.354
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Kohsuke Kawaguchi <kk@kohsuke.org>
 dpkg-source -Zgzip --before-build .
 fakeroot debian/rules clean
dh clean
   dh_clean
 debian/rules build-indep
dh build-indep
   dh_update_autotools_config -i
   dh_autoreconf -i
   create-stamp debian/debhelper-build-stamp
 fakeroot debian/rules binary-indep
dh binary-indep
   dh_testroot -i
   dh_prep -i
   dh_installdirs -i
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 2: usr/share/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 5: var/lib/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 9: var/log/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 12: var/cache/jenkins: not found
dh_installdirs: warning: debian/jenkins.dirs is marked executable but does not appear to an executable config.
dh_installdirs: warning:
dh_installdirs: warning: If debian/jenkins.dirs is intended to be an executable config file, please ensure it can
dh_installdirs: warning: be run as a stand-alone script/program (e.g. "./debian/jenkins.dirs")
dh_installdirs: warning: Otherwise, please remove the executable bit from the file (e.g. chmod -x "debian/jenkins.dirs")
dh_installdirs: warning:
dh_installdirs: warning: Please see "Executable debhelper config files" in debhelper(7) for more information.
dh_installdirs: warning:
dh_installdirs: error: debian/jenkins.dirs (executable config) returned exit code 127
make: *** [debian/rules:4: binary-indep] Error 25
dpkg-buildpackage: error: fakeroot debian/rules binary-indep subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -Zgzip -A failed
+ rm -rf /tmp/tmp.GaXJspqUF9
make: *** [Makefile:40: target/debian/jenkins_2.354_all.deb] Error 29

usr/share/jenkins is maybe missing a / but /usr/share/jenkins exists.
var/lib/jenkins is maybe missing a / and /var/lib/jenkins does not exist.
var/log/jenkins is maybe missing a / and /var/log/jenkins does not exist.
var/cache/jenkins is maybe missing a / and /var/cache/jenkins does not exist.

Maybe there is something obvious that I missed. Can anyone help?
Thanks in advance.

I can’t see anything obvious.

1 Like

Is there any reason your using cygwin instead of wsl?

This feels like the shell your using doesn’t have the flag your using

If you run it with debug mode bash -x ./prep.sh does it tell you what like is failing?

1 Like

Yes, the console that goes with WSL doesn’t work well for me, that’s why I’m sticking with cygwin.
I have gotten rid of the end of lines problem (i.e. ./prep.sh : invalid option.) by using a .gitattributes file (PR accepted).
The ./prep.sh now works
I still have problems with make though…

/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 2: usr/share/jenkins: not found`
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 5: var/lib/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 9: var/log/jenkins: not found
/tmp/tmp.GaXJspqUF9/debian/jenkins.dirs: 12: var/cache/jenkins: not found
dh_installdirs: warning: debian/jenkins.dirs is marked executable but does not appear to an executable config.
dh_installdirs: warning:
dh_installdirs: warning: If debian/jenkins.dirs is intended to be an executable config file, please ensure it can
dh_installdirs: warning: be run as a stand-alone script/program (e.g. "./debian/jenkins.dirs")
dh_installdirs: warning: Otherwise, please remove the executable bit from the file (e.g. chmod -x "debian/jenkins.dirs")
dh_installdirs: warning:
dh_installdirs: warning: Please see "Executable debhelper config files" in debhelper(7) for more information.
dh_installdirs: warning:
dh_installdirs: error: debian/jenkins.dirs (executable config) returned exit code 127
make: *** [debian/rules:4: binary-indep] Error 25

Thanks. :pray:

I have now progressed.
I still can’t build .deb package with the same error, but I can build for suse and rpm.

find target -name "*"
target
target/rpm
target/rpm/jenkins-2.355-1.1.noarch.rpm
target/suse
target/suse/jenkins-2.355-1.2.noarch.rpm

I have looked at the Jenkinsfile variables (thanks @dduportal for the hint) and have set them in the shell:

WORKSPACE=$(pwd)
BUILDENV=${WORKSPACE}/env/test.mk
BRANDING_DIR=${WORKSPACE}/branding
BRAND=${WORKSPACE}/branding/jenkins.mk
GPG_FILE=${WORKSPACE}/credentials/sandbox.gpg
GPG_KEYNAME="Bogus Test"
GPG_PASSPHRASE=s3cr3t
GPG_PASSPHRASE_FILE=${WORKSPACE}/credentials/test.gpg.password.txt
WAR=${WORKSPACE}/jenkins.war
MSI=${WORKSPACE}/jenkins.msi
RELEASELINE=-experimental

I will now have a look at jenkins-ci build logs to see if I’m missing something.

In the meantime, I have managed to build without a glitch with a Ubuntu VM supplied by Oracle Free Tier (Linux instance-20220510-1550 5.13.0-1030-oracle #35~20.04.1-Ubuntu SMP Wed May 25 23:16:51 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux).

I have also built in a Ubuntu VM (Linux a0563e95906c 5.15.0-35-generic #36-Ubuntu SMP Sat May 21 02:24:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux) without any issue.

While I was there, I tried to build on another Oracle Free Tier machine (Linux gitlab-runner 5.13.0-1036-oracle #43~20.04.1-Ubuntu SMP Tue Jun 14 01:06:54 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux) but got the infamous exec /usr/local/bin/jenkins-agent: exec format error, which means the image is not multi-arch (I will have a look at it, but that’s for another day).

I have also tried with WSL2 running Debian, and I have the very same error I have with cygwin.
So… It definitely looks like the error is tightly linked to Windows.