Version controlling Jenkins config: Help defining a `.gitignore` that minimizes the git repo size

kaushalmodi · July 15, 2022, 1:16pm

Hello,

I was recently burned out when a Project matrix misconfig got me locked out of my own Jenkins instance and I then needed to set up all the jobs from scratch.

So now I am attempting to version control the whole Jenkins config – essentially creating a Jenkins image that I can simply git clone and get the exact Jenkins server up and running in seconds.

I have achieved that mostly but at the expense of committing the large plugins/ directory to git. I came across posts like these which suggest not to commit plugins/ to git: Is there a way to keep Hudson / Jenkins configuration files in source control? - Stack Overflow

But if I do that, I get errors like these on attempting the start the Jenkins server after cloning that repo:

2022-07-15 13:07:06.082+0000 [id=31]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading global config
com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.security.ProjectMatrixAuthorizationStrategy
        at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:81)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:411)
        at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:46)
        at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:125)
        at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:47)
        at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:521)
        at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:346)
Caused: jenkins.util.xstream.CriticalXStreamException:
---- Debugging information ----
cause-exception     : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message       : hudson.security.ProjectMatrixAuthorizationStrategy
class               : hudson.model.Hudson
required-type       : hudson.model.Hudson
converter-type      : hudson.util.RobustReflectionConverter
path                : /hudson/authorizationStrategy
line number         : 12
version             : not available
-------------------------------
        at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:381)
        at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
        at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
        at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1421)
        at hudson.util.XStream2.unmarshal(XStream2.java:189)
        at hudson.util.XStream2.unmarshal(XStream2.java:160)
        at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1399)
        at hudson.XmlFile.unmarshal(XmlFile.java:196)

Ideally, I would like to prevent committing the entire plugins/ directory as it mainly consists of binary files.
Is there a minimal set of plugin-related files I can track in git such that, when I do the first Jenkins start, it will auto-download the plugins of the tracked versions and start the server?

My Jenkins `.gitignore`

Here’s my .gitignore:

# -*- conf -*-
# References:
# - https://stackoverflow.com/a/4695615/1219634
# - https://stackoverflow.com/a/38297609/1219634

# Cache
caches/

# Fingerprint records
fingerprints/

# Working directories
workspace/

# Secret files
secrets/
secret.*
*.enc
*.key
users/

# State files
*.state

# Updates
updates/

# Extracted jenkins.war
war/

# Job state files
builds/
nextBuildNumber

# Indexing
indexing/

# Log files
logs/
*.log

# Help files
help/

# Miscellaneous litter
*.bak
.nfs*
.lastStarted
.owner
.timestamp
queue.xml
downloads/
jenkins.war

Version info

Jenkins: 2.346.2

Java:

openjdk 11.0.2 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

OS: CentOS 7.6.1810

MarkEWaite · July 15, 2022, 2:17pm

The plugin installation manager tool will read a file (either a plain text file like plugins.txt or a YAML file) and download the plugins that are specified in that file if they are not already downloaded. It does its job very well.

I think you should prefer to have the plugins locally somehow rather than delaying the startup of your Jenkins controller while it downloads plugins. Many installations were affected when the Jenkins update center was down and they could not download their plugins on each startup of their controller. By keeping a local copy of the plugin files (how ever you choose to do that), you avoid making controller startup depend on the download of the plugins.

One technique for that is to define a Docker container image that includes the plugins inside the image. Build the container image and download the plugins during the container build process. Because the container includes the plugins, it starts quickly without needing to download the plugins.

kaushalmodi · July 15, 2022, 2:25pm

Thanks Mark. Is there a way to auto-update the plugins.txt each time I update the plugins from the Jenkins GUI?

I think you should prefer to have the plugins locally somehow rather than delaying the startup of your Jenkins controller while it downloads plugins.

I won’t need to do a fresh clone of this repo frequently… only when somethings breaks and I need to trace what what changed. If I can version-control only that auto-updated plugins.txt, the solution will be optimal.

Right now, the plugins dir is around 300MB containing mainly binary files. If I keep on committing the updates in that same repo, very soon my repo will become a couple of GB in size!

One technique for that is to define a Docker container image that includes the plugins inside the image.

Sorry, that falls out of my knowledge scope. I will need first figure out how to install docker at work and then dive into that rabbit hole.

I am trying to keep things simple… it looks like the plugins.txt you mentioned will allow requirements.txt + pip install kind of flow (Python package dependency installation), and I will need to maintain only a light-weight plugins.txt file in the git repo instead of the large plugins/ dir.

Question: Looking for a flow that auto-updates the plugins.txt each time I update the plugins from Jenkins Plugins Manager.

kaushalmodi · July 15, 2022, 2:28pm

Just to add more context to what I said above… I just updated 2 plugins and here’s how that looks in a git commit:

I would be nice to instead see only 2 lines change in a plugins.txt plain text file.

MarkEWaite · July 15, 2022, 2:31pm

I’m not aware of anything that does that, though I can easily envision you doing that with a shell script that runs periodically.

I use the following two commands with my configuration:

$ ./jenkins-plugin-cli.sh --jenkins-version 2.346.2 -d ref/plugins -f plugins.txt --no-download --available-updates -o txt > x && mv x plugins.txt
$ ./jenkins-plugin-cli.sh --jenkins-version 2.346.2 -d ref/plugins -f plugins.txt

The first line asks the Jenkins update center for available updates and replaces the local plugins.txt with the revised version (including updates). The second line downloads the plugin versions that are specified in the new plugins.txt file.

kaushalmodi · July 15, 2022, 2:38pm

Thanks. this might be what I need. I will look into this once I get time to play with jenkins-plugins-manager.jar.

For my reference: docker-lfs/jenkins-plugin-cli.sh at 673fc4f87f158ebf34124673110bba93bea6e7d1 · MarkEWaite/docker-lfs · GitHub

MarkEWaite · July 15, 2022, 2:43pm

If you add all the files in the plugins folder to a .gitignore and then allow *.jpi files in that directory, you won’t waste the space of placing the unpacked plugin contents into your git repository. However, that would still place the plugin binary files in your repository. That will cause the repository to grow quickly.

I needed to test the git large file storage facility, so my environment uses git LFS to store plugin binary files. That’s unusual. Most administrators only store the plugins.txt file in the git repository.

kaushalmodi · July 15, 2022, 2:44pm

I tried that approach too, but I got a similar crash. In that attempt, I had retained the whole plugins/ directory and removed only the *.jar files.

kaushalmodi · July 15, 2022, 2:48pm

I made a quick attempt at using jenkins-plugins-manager and it didn’t work. I will need to read up more on this …

Failed attempt:

Downloaded https://github.com/jenkinsci/plugin-installation-manager-tool/releases/download/2.12.8/jenkins-plugin-manager-2.12.8.jar
touch plugins.txt
java -jar jenkins-plugin-manager-2.12.8.jar --jenkins-version 2.346.2 -d webroot/plugins/ -f plugins.txt --no-download --available-updates -o txt > x && mv x plugins.txt

… plugins.txt is empty

I think it’s empty because I just updated all the plugins. The --available-updates switch will be looking only for updates.

I will read up to see if there’s a command that reads through my existing plugins dir webroot/plugins and create a plugins.txt based on that.

This comes close: java -jar jenkins-plugin-manager-2.12.8.jar --jenkins-version 2.346.2 -d webroot/plugins/ -l. But the output has too much noise… there might be another switch with simply outputs a list of installed plugins, and not headers like

Bundled plugins:
-none-

All requested plugins:
-none-

Plugins that will be downloaded:
-none-

Resulting plugin list:

I am documenting this for my self so that I know where to continue this effort from the next time I find time.

MarkEWaite · July 15, 2022, 3:19pm

Support output format for list option by cronik · Pull Request #448 · jenkinsci/plugin-installation-manager-tool · GitHub describes the arguments (added in 2.12.8) to generate a plugins.txt file from the contents of an existing plugins folder. Your command line is close, but you need to redirect stdout to a file so that the informational messages are not intermixed with the generated plugins.txt file.

kaushalmodi · July 15, 2022, 4:04pm

Thanks! I am now able to generate a plugins.txt.

As a next step, I tried removing the populated plugins/ dir and populating it again using jenkins-plugin-manager, but it failed with this same error for each plugin:

Unable to resolve plugin URL https://updates.jenkins.io/latest/Resultingpluginlist.hpi, or download plugin Resultingpluginlist to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/Resultingpluginlist.hpi, or download plugin Resultingpluginlist to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/Resultingpluginlist.hpi, or download plugin Resultingpluginlist to file: status code: 404, reason phrase: Not Found
Downloading from mirrors failed, falling back to https://archives.jenkins.io/
Unable to resolve plugin URL https://archives.jenkins.io/plugins/Resultingpluginlist/latest/Resultingpluginlist.hpi, or download plugin Resultingpluginlist to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ace-editor1.1.hpi, or download plugin ace-editor1.1 to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ace-editor1.1.hpi, or download plugin ace-editor1.1 to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ace-editor1.1.hpi, or download plugin ace-editor1.1 to file: status code: 404, reason phrase: Not Found
Downloading from mirrors failed, falling back to https://archives.jenkins.io/
Unable to resolve plugin URL https://archives.jenkins.io/plugins/ace-editor1.1/latest/ace-editor1.1.hpi, or download plugin ace-editor1.1 to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ant475.vf34069fef73c.hpi, or download plugin ant475.vf34069fef73c to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ant475.vf34069fef73c.hpi, or download plugin ant475.vf34069fef73c to file: status code: 404, reason phrase: Not Found
Unable to resolve plugin URL https://updates.jenkins.io/latest/ant475.vf34069fef73c.hpi, or download plugin ant475.vf34069fef73c to file: status code: 404, reason phrase: Not Found

Is this because of what you mentioned earlier?

Here’s the entire plugins.txt:

Resulting plugin list:
ace-editor 1.1
ant 475.vf34069fef73c
antisamy-markup-formatter 2.7
apache-httpcomponents-client-4-api 4.5.13-1.0
atlassian-bitbucket-server-integration 3.2.2
authentication-tokens 1.4
authorize-project 1.4.0
bitbucket 223.vd12f2bca5430
bitbucket-push-and-pull-request 2.8.2
bootstrap4-api 4.6.0-5
bootstrap5-api 5.1.3-7
bouncycastle-api 2.26
branch-api 2.1046.v0ca_37783ecc5
build-timeout 1.21
build-timestamp 1.0.3
caffeine-api 2.9.3-65.v6a_47d0f4d1fe
checks-api 1.7.4
cloudbees-bitbucket-branch-source 773.v4b_9b_005b_562b_
cloudbees-folder 6.729.v2b_9d1a_74d673
command-launcher 84.v4a_97f2027398
credentials 1139.veb_9579fca_33b_
credentials-binding 523.vd859a_4b_122e6
display-url-api 2.3.6
durable-task 496.va67c6f9eefa7
echarts-api 5.3.3-1
email-ext 2.89
emailext-template 1.4
envfile 1.2
envinject 2.866.v5c0403e3d4df
envinject-api 1.199.v3ce31253ed13
external-monitor-job 191.v363d0d1efdf8
font-awesome-api 6.1.1-1
git 4.11.3
git-client 3.11.0
git-server 1.11
gradle 1.39.4
handlebars 3.0.8
handy-uri-templates-2-api 2.1.8-22.v77d5b_75e6953
jackson2-api 2.13.3-285.vc03c0256d517
javadoc 217.v905b_86277a_2a_
javax-activation-api 1.2.0-3
javax-mail-api 1.6.2-6
jaxb 2.3.6-1
jdk-tool 1.5
jjwt-api 0.11.5-77.v646c772fddb_0
job-dsl 1.80
jquery-detached 1.2.1
jquery3-api 3.6.0-4
jsch 0.1.55.2
junit 1119.1121.vc43d0fc45561
ldap 2.10
lockable-resources 2.15
mailer 414.vcc4c33714601
mapdb-api 1.0.9.0
matrix-auth 3.1.5
matrix-project 772.v494f19991984
mercurial 2.16.2
mina-sshd-api-common 2.8.0-21.v493b_6b_db_22c6
mina-sshd-api-core 2.8.0-21.v493b_6b_db_22c6
momentjs 1.1.1
okhttp-api 4.9.3-105.vb96869f8ac3a
p4 1.12.4
pam-auth 1.8
pipeline-build-step 2.18
pipeline-graph-analysis 195.v5812d95a_a_2f9
pipeline-groovy-lib 593.va_a_fc25d520e9
pipeline-input-step 449.v77f0e8b_845c4
pipeline-milestone-step 101.vd572fef9d926
pipeline-model-api 2.2097.v33db_b_de764b_e
pipeline-model-definition 2.2097.v33db_b_de764b_e
pipeline-model-extensions 2.2097.v33db_b_de764b_e
pipeline-rest-api 2.24
pipeline-stage-step 293.v200037eefcd5
pipeline-stage-tags-metadata 2.2097.v33db_b_de764b_e
pipeline-stage-view 2.24
plain-credentials 1.8
plugin-util-api 2.17.0
popper-api 1.16.1-3
popper2-api 2.11.5-2
resource-disposer 0.19
scm-api 608.vfa_f971c5a_a_e9
script-security 1175.v4b_d517d6db_f0
snakeyaml-api 1.30.2-76.vc104f7ce9870
ssh-credentials 291.v8211e4f8efb_c
ssh-slaves 1.834.v622da_57f702c
sshd 3.242.va_db_9da_b_26a_c3
structs 318.va_f3ccb_729b_71
timestamper 1.18
token-macro 293.v283932a_0a_b_49
trilead-api 1.67.vc3938a_35172f
variant 1.4
windows-slaves 1.8.1
workflow-aggregator 590.v6a_d052e5a_a_b_5
workflow-api 1188.v0016b_4f29881
workflow-basic-steps 969.vc4ec3e4854b_f
workflow-cps 2746.v0da_83a_332669
workflow-durable-task-step 1174.v73a_9a_17edce0
workflow-job 1207.ve6191ff089f8
workflow-multibranch 716.vc692a_e52371b_
workflow-scm-step 400.v6b_89a_1317c9a_
workflow-step-api 625.vd896b_f445a_f8
workflow-support 833.va_1c71061486b_
ws-cleanup 0.42

MarkEWaite · July 15, 2022, 4:13pm

The plugin identifier and the plugin version need to be separated by a ‘:’ character rather than a ’ ’ space character in the file.

kaushalmodi · July 15, 2022, 4:13pm

@MarkEWaite The fix for above was to generate the plugins.txt using the --output TXT to the jenkins-plugin-manager.

Now testing out installation of plugins using that file …

kaushalmodi · July 15, 2022, 4:26pm

Success!!

Script to update `plugins.txt`

#!/usr/bin/env bash

# Update a text file containing the installed plugins in
# webroot/plugins/ and their versions.

set -euo pipefail # http://redsymbol.net/articles/unofficial-bash-strict-mode
IFS=$'\n\t'

jpm_repo_url="https://github.com/jenkinsci/plugin-installation-manager-tool"
jenkins_plugins_dir="webroot/plugins/"
plugins_file="plugins.txt"

# Initialize variables
jpm_version="2.12.8"
debug=0

tmp_dir="/tmp/${USER}/"
while [ $# -gt 0 ]
do
    case "$1" in
        "-v"|"--version" ) shift
                           jpm_version="$1";;

        "-D"|"--debug" ) debug=1;;
    esac
    shift # expose next argument
done

main () {
    if [[ ${debug} -eq 1 ]]
    then
        echo "Debug mode"
    fi

    jpm_jar_path="${tmp_dir}/jenkins-plugin-manager.${jpm_version}.jar"

    if [[ ! -f "${jpm_jar_path}" ]]
    then
        curl -RL "${jpm_repo_url}"/releases/download/"${jpm_version}"/jenkins-plugin-manager-"${jpm_version}".jar \
             --create-dirs -o "${jpm_jar_path}"
    fi

    echo "Jenkins Plugins Manager version:"
    java -jar "${jpm_jar_path}" --version

    echo "Updating plugins.txt .."
    java -jar "${jpm_jar_path}" \
         --plugin-download-directory "${jenkins_plugins_dir}" \
         --output TXT \
         --list > "${plugins_file}"
}

main

Script to install plugins and start Jenkins server

#!/usr/bin/env bash

# Download the war file for the Jenkins version specified in
# ${jenkins_version} and start Jenkins server.

set -euo pipefail # http://redsymbol.net/articles/unofficial-bash-strict-mode
IFS=$'\n\t'

# Initialize variables
port=8080
jenkins_version="2.346.2"
debug=0

jpm_version="2.12.8"
plugins_file="plugins.txt"

tmp_dir="/tmp/${USER}/"
while [ $# -gt 0 ]
do
    case "$1" in
        "-p"|"--port" ) shift
                        port="$1";;
        "-v"|"--version" ) shift
                           jenkins_version="$1";;

        "-D"|"--debug" ) debug=1;;
    esac
    shift # expose next argument
done

jenkins_war_path="${tmp_dir}/jenkins.${jenkins_version}.war"

# https://community.jenkins.io/t/version-controlling-jenkins-config-help-defining-a-gitignore-that-minimizes-the-git-repo-size/3036
update_plugins () {
    jpm_repo_url="https://github.com/jenkinsci/plugin-installation-manager-tool"
    jenkins_plugins_dir="webroot/plugins/"

    jpm_jar_path="${tmp_dir}/jenkins-plugin-manager.${jpm_version}.jar"

    if [[ ! -f "${jpm_jar_path}" ]]
    then
        curl -RL "${jpm_repo_url}"/releases/download/"${jpm_version}"/jenkins-plugin-manager-"${jpm_version}".jar \
             --create-dirs -o "${jpm_jar_path}"
    fi

    echo "Updating ${jenkins_plugins_dir} .."
    java -jar "${jpm_jar_path}" \
         --war "${jenkins_war_path}" \
         --jenkins-version "${jenkins_version}" \
         --plugin-download-directory "${jenkins_plugins_dir}" \
         --plugin-file "${plugins_file}"
}

main () {
    if [[ ${debug} -eq 1 ]]
    then
        echo "Debug mode"
    fi

    if [[ ! -f "${jenkins_war_path}" ]]
    then
        curl -RL https://get.jenkins.io/war-stable/"${jenkins_version}"/jenkins.war \
             --create-dirs -o "${jenkins_war_path}"
    fi

    if [[ -f "${plugins_file}" ]]
    then
        update_plugins
    fi

    echo "Jenkins version:"
    java -jar "${jenkins_war_path}" --version

    echo "Starting server .."
    JENKINS_HOME="$(pwd)/webroot" java -jar "${jenkins_war_path}" --httpPort="${port}"
}

main

`.gitignore`

# -*- conf -*-
# References:
# - https://stackoverflow.com/a/4695615/1219634
# - https://stackoverflow.com/a/38297609/1219634

# Cache
caches/

# Fingerprint records
fingerprints/

# Working directories
workspace/

# Secret files
secrets/
secret.*
*.enc
*.key
users/

# State files
*.state

# Updates
updates/

# Plugins
plugins/

# Extracted jenkins.war
war/

# Job state files
builds/
nextBuildNumber

# Indexing
indexing/

# Log files
logs/
*.log

# Help files
help/

# Miscellaneous litter
*.bak
.nfs*
.lastStarted
.owner
.timestamp
queue.xml

kaushalmodi · July 15, 2022, 4:31pm

This is great! Now after a fresh clone, my jenkins repo is < 1MB in size!

_Now I just need to purge all the old .jar and other binary files from the .git/

kaushalmodi · July 15, 2022, 6:32pm

@MarkEWaite Thanks for all your help.

I am wrapping up this thread with a link to my repo that includes everything I have learned here: GitHub - kaushalmodi/jenkins-minimal: Minimal Jenkins config repo

Topic		Replies	Views
Error cloning remote repo 'origin' intermittent issue in Jenkins-2.452.3 Ask a question	1	783	July 25, 2024
Git plugin error from last 2 days Using Jenkins question	2	1462	February 29, 2024
Cannot Resolve Class Exception: hudson. security. Global Matrix Authorization Strategy Using Jenkins question	7	11161	February 2, 2024
Jenkins will not start after I accidently updated all plugins Using Jenkins	1	96	November 23, 2024
Jenkins failing to load all plugins after server reboot. ( not able to access web panel because of stack trace) Ask a question question	3	4522	May 25, 2023

Version controlling Jenkins config: Help defining a `.gitignore` that minimizes the git repo size

My Jenkins .gitignore

Version info

Script to update plugins.txt

Script to install plugins and start Jenkins server

.gitignore

Related topics

My Jenkins `.gitignore`

Script to update `plugins.txt`

`.gitignore`