How to Securing Jenkins Config As Code?

Community
1

Hey, folks
I’am planning on migrating our old Jenkins setup to newer version along with new strategy.

Old setup:

  • Majority of the jobs are connect to remote server and execute a shell script to perform the build actions like ( git pull, rsync to production doc-root etc.) on multiple servers serially.

  • For user management and access we use Auth plugins along with project based matrix.

Why are we upgrading?

  • Setup is stuck with old dependencies, and we are unable to upgrade Jenkins version and plugin versions likely due to broken dependencies on some job.

What we are trying to achieve with the new setup?

  • Need a flexible setup.

  • Quick disaster recovery.

  • More automated setup.

  • Make setup complaint with compliance and security policies.

In order to achieve above: We are trying to explore the the approach of Jenkins Config As Code(jcasc).

What we have achieved so far :

Automation:
A fully automated docker-based setup.

  • DR:
    We are quickly able spin a new Jenkins instance whenever we want by just running a single docker-compose.

Biggest advantages is we are able to replicate same setup as it is on new server without worrying about missing x/y setting.

However we facing concerns regarding the following: ( Some of these are explored and some are yet to be explored.)

  • Credentials:
    1. How to securely store the user credentials with (jcasc) since all the config is stored in plain-text?
    2. For config management we cannot directly commit config to git or any other SCM, as passwords are still in plain text.
    3. Alternative to this is we can use encoded password evoked from variables on Jenkins config file, however locally stored passwords can still be decrypted.

(There is also a possibility to use key vault for credentials, yet to explore that.)

  • Security / Auth Tokens:
    We’ve faced major challenge in creating persistent Auth tokens on Jenkins as whenever we refresh the setup old token would be useless and would break major pipelines.

In a nutshell most of the flexibility part for the setup has been achieved but we stuck with security part of it, anyone who has faced these challenges kindly guide use in right direction.

Thank You.

Regards

2

All three of your questions are addressed with various alternatives in the configuration as code documentation for secret management.

An example is available in a README

As an example, I have some personal credentials saved on my computer (like ~/.ssh/id_ed25519) and the container images that I build are able to read that file and make it a Jenkins credential with the following configuration as code entry:

credentials:
  system:
    domainCredentials:
    - credentials:
      - basicSSHUserPrivateKey:
          description: "${USER} ED25519 private key"
          id: "${USER}-ed25519-private-key"
          privateKeySource:
            directEntry:
              privateKey: "${readFile:${HOME}/.ssh/id_ed25519}"
          scope: SYSTEM
          username: "${USER}"

That works for me because I build my container images on my local computer and that local computer always has the file ~/.ssh/id_ed25519.

3

In our setup (which is older than Casc) we use ansible for configuration. files which contain secrets, e.g. the credentials.xml but also the controller key of Jenkins is stored encrypted (we have named instances so each instance has its own key but we can easily spin it up anywhere and we’re sure all secrets are readable).
When starting the service we first run ansible which creates the complete config of Jenkins with a few core jobs that are needed everywhere. Then we have our own Job generator running at startup that creates/updates all jobs with an instance specific job base.
Even with 100% data loss we’re able to recover the instance within 1h (except for any job that was running), but we never needed to do this.

4

Thank you for your reply, I’ve already tried couple of those links shared.
been occupied with some stuff was not able to work much on it hopefully I will be able to push this in upcoming week.

5

well the Instance I am working on has roughly 300+ active build jobs, and should have around 80-100 active users who make their builds from the instance. I don’t know how updated credentials can be managed in this dynamic setup where new jobs and users keep on adding almost every week.
Number of users may be bit on stable side, perhaps there is no such discount in growth for number of jobs. Let me try to wack my brain a little on this.

6

When your instance is so dynamic you need to use backups for the things that change frequently. Those can be stored in git. Just don’t store the controller key in git unencrypted. All other secrets in Jenkins are encrypted in the xml files based on the controller key.
You can still use CasC for things that don’t change often (e.g. plugin config)
For user management it seems you’re using the jenkins internal user database. It would be better to use here something else, e.g. ldap, Active directory, SAML or something else connected to a corporate idp.

In our setup we store the complete users directory in github (updated once per hour). As the config.xml of a user contains a timestamp we set that to 0 so only real user config changes are persisted.