Running Jenkins from inside a container and passing directories/files to it's child containers is impossible

Using Jenkins
1

For example, you have a job called my_job and you want to check out your repository to its workspace at /var/jenkins_home/workspaces/my_job and then scan it using some kinda containerized linting tool. That seems like a pretty normal and valid use case but it does not work.

There are a couple of ways to set up a Jenkins container so it can create additional containers.

  1. You can install the docker client into the Jenkins container. You can then pass in the docker socket from the host machine into the jenkins container. This will work for both creating containers and passing files/folders to the container but it has one major issue. When you go to pass the my_job workspace to the linting container you wont do this docker run -v /var/jenkins_home/my_job:/files_to_lint linter. That is what you might try at first but that will not work. On the container /files_to_lint will be empty because you are currently in a container and it’s filesystem cant be bind mounted into another container. So can bind mount the jenkins container to the host, like docker run -v /tmp/jenkins_container_home:/var/jenkins_home jenkins/jenkins-lts. Then when you start your linter you would run docker run -v /tmp/jenkins_container_home/workspaces/my_job:/files_to_lint linter. That would work but its pretty confusing at first and hard to explain to others. You also wont always have the option with third party tools to have it specify some strange path, instead of the current work directory.
  2. The other option is to install a docker daemon into the Jenkins container. This is extremely hard to do and its very brittle. I had this working for about 4 months and then it broke when either ubuntu or docker updated. I think its when docker version 23 came out. I have since tried getting docker 23 to work have spent hours and no luck. When this works its one of the better options.
  3. You can run a separate docker daemon like docker:dind and then connect your jenkins container to it via some environment variables like:
environment:
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1

This seems to be the recommended best practice for running docker. It’s similar to how gitlab cicd works. The problem I have had with this is that no matter what I do. When I try mounting a workspace into a child container then the workspace shows up as empty. Which makes sense, This is similar to the problems with #1. Where the path to the files on the daemon is different than what they are locally.

I’m not saying you cant build a docker image and then push it. I’m just saying you can’t reliably/easily mount your code into some other container.

The only way I have gotten this to work is to run Jenkins outside of a container on the host machine/vm.

I have been fighting this for 3 years off and on. Am I crazy? Is there some easier way that I somehow don’t know about?