Hi all,
I am building a CI system with Jenkins at its core. I am trying to have as much as possible “as code” in git. The various components of the CI system are saved in different git repositories (gitlab).
As I update those components I am wondering what would be a good way to version the components and keep them in sync (eg: to allow easy rollback if needed).
For context, here are the 5 main repositories:
jenkins (docker, config as code, plugin list)
jenkins-nodes (ansible scripts to setup the nodes)
ci-scripts (helpers scripts used by pipelines)
ci-pipelines (1 repository per software platform)
robotframework-tests (for product test.)
At first I was updating several repositories together. My release process was to tag each commit with the same tag version (e.g. currently all are at 1.1) and create a release note for each in gitlab (with links to the other commits). But this is rather admin heavy and potentially error prone. Also it would mean re-tagging the same commits when only one repository is being updated.
Now I am looking if some other options would be better.
For example using git-repo to manage the different repositories with a manifest file. https://gerrit.googlesource.com/git-repo
Or to invent a better tagging/versioning system.
Before I start spending more time on that, it would be great if others would want to share their experience.
I appreciate this is not strictly a Jenkins question, apologies if this is not appropriate for this forum.
TLDR: how do you organize & version your different CI components, in order to keep the history across components, and to keep them in sync to rollback easily if needed?
Technically, one approach in the confines of Git could be to use yet another repo as a “dispatcher”, with defined git submodules. Those definitions refer to known repository URLs, and bolt down specific commits of your “real” repos as the data tracked in the dispatcher. You can tag, branch, and otherwise manipulate this dispatcher as you would any other git repo - e.g. to track multi-repo release branches separately from the main development trunk (and possibly referring to different populations of those other components).
Usually there is no need to release and tag everything together.
We also have several repositories:
Jenkins image (builds a Jenkins docker image with plugins)
Jenkins configuration (ansible scripts that creates the configuration of Jenkins, refers to the images created in 1), this also includes generating some admin jobs
Jenkins agents (just some yaml files defining the agents in Jenkins, there is an admin job that will react on changes in this repo and creates new agents or adjusts labels)
Agent setup with Chef
job configuration (We have a self written job generator that will generate our thousands of jobs)
pipeline libraries
systemd scripts to start Jenkins
There can be dependencies between the repos, e.g. when the pipeline library needs a certain plugin, but it doesn’t happen so often that a new plugin is needed. More often I need to adjust the pipeline and I want such changes to become active without the need to restart Jenkins.
But for example we don’t have a real dependency from the Jenkins agents repo to the Jenkins image. What we install on the agents is basically just the basics so we can run java and git, have proper user setup and install required tools to build our software.
The job configuration is also only loosely coupled to the Jenkins image and configuration. Some jobs configs have been created when we were still on Jenkins 1.x, we haven’t updated them and they still work on 2.375.x
I think it is essential that you can easily setup a test system where you check a new Jenkins version, changes to the configuration, pipelines, jobs and scripts.
@jimklimov , thank you for the suggestion. I had heard of git submodule but your explanation and examples made that clearer.
@mawinter69 , thank you for your reply. I see what you mean by loose coupling. I guess somehow what you are saying is that if I test changes properly then I should not need too precise rollback?
I have already set up a test environment in which I do all the development and testing, so I should be fine there.
I will give all that more thoughts but currently leaning to versioning and releasing the various components independently of each other, and if some version are strongly coupled I would state it in the release notes for future reference.