Recently, I encountered an issue in my Jenkins practice: we use a Kubernetes-based controller-agent distributed architecture to support continuous integration for projects. I created a job named, for example, build-job-for-grep , in which I clone the code repository and then scan the diff files compared to the main branch using grep for specific patterns to inspect incremental code security. This job often runs concurrently.
Yesterday, an unexpected incident occurred: two jobs executed simultaneously, with build numbers #100 and #101. These two jobs pulled two different code repositories, but the grep results were exactly the same, meaning one job obtained the results of the other—a very bizarre situation.
As mentioned earlier, we use Kubernetes container resources for builds. At that time, these two jobs created two separate Pods, which coincidentally ran on the same Node. Another important factor is that when these two jobs executed the grep command, the paths in their respective Pods were exactly the same: /home/jenkins/workspace/build-job-for-grep/increment/
However, I did not set up any shared mount for the job’s workspace directory.
- I would like to consult: what could be the reason for such a result?
- Also, does this indicate risks in concurrent scenarios for the same Jenkins job?
- How can we avoid such concurrency risks?