Jenkins pipeline optimization - how to build images selectively based on code change [details inside]

Jenkins setup:

Jenkins: 2.462.1
OS: Linux - 6.5.0-1017-aws
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

I don’t think plugins are required at this point of time.

So here is the situation:

Current status:

    • we are using declarative pipeline, one big Jenkinsfile, for our repo with “solutions” (I will call them A, B and C) - imagine them as logical grouping.
  • directories are “divided” not per solutions, but per prog languages. Note: each solution consist of “services” (I will call them A1, A2, AN… B1, B2, BN… and C1, C2, CN) and they are not tied to one specific language - but these are physical groups (each service is represented by a bunch of code files in GH repo and as container process in production env). Note: Each group of these services forms one solution.
  • if code inside one directory is modified, the CI part (lint, test, build, push to dockerhub) defined in Jenkinsfile is triggered for ALL services, but only inside that directory. If code inside two directory is modified, then ALL services from both of these directories are affected in pipeline, etc. You get the point.
  • CD part, for which we use ArgoCD, deploys ALL services from ALL directories, regardless from which directory the code was modified. Of course, Jenkins is the one to “change” image tags (in yaml files that ArgoCD looks at) ONLY for the new services affected by CI.

So what we are aiming for?

  • declarative or scripted pipeline, we don’t care for now, but since we are hitting “Method code too large!” often, maybe scripted. Note: we are not restricted to have only 1 Jenkinsfile… we just happen to have 1 huge file.
  • directories still “divided” per prog languages. TBH, we have another logical grouping inside their subdirectories and it will take too much effort to re-group codebase.
  • When the code inside one or more directories is modified, the CI part defined in Jenkinsfile is triggered ONLY for solution(s), from which the service codebase is changed.
  • CD part, for which we use ArgoCD, deploys services ONLY from modified solution, (still) regardless from which directory the code was modified. Of course, Jenkins is the one to “change” image tags (in yaml files that ArgoCD looks at) ONLY for the new services affected by CI.

So keeping aside the thought of revamping the repository, is there any mechanism from the Jenkins side of things to achieve what we are looking for?

Summary:
Right now, if there is a code change only in directory called “java”, CI part is affecting services A1, A2, B2, B3, C1. If there is a code change only in directory called “go” CI part is affecting services A3, B1, C2, C3. Anyway, after CI part is over, what is being deployed is: A1, A2, A3, A4, B1, B2, B3, B4, C1, C2, C3, C4.
In future, regardless of which directory the code change is made, if code modification happens for a service belonging to solution A, then only A1, A2, A3, A4 should be run inside CI pipeline AND deployed.

Is there any mechanism from the Jenkins “side of things” to achieve what we are looking for?

tl;dr trying to selectively build and deploy services for which there was a code change and not for all, in a monolithic codebase.

Yes, the root pipeline Jenkinsfile does have a limit to how large it can be. Shared Pipeline Libraries have separate limits to them so, if your pipeline has methods handling your various build/deployment cases or it can be turned into methods, you can move them to a shared pipeline library and call methods from your pipeline.

Your starting point for this will be in Extending with Shared Libraries

The git plugin allows to define exclude and include patterns e.g. when doing polling but that might also be available in multi branch jobs I think.
So you can setup a job for each solution and define corresponding patterns so the job only starts when a file for that solution is changed.
Don’t know if that is possible for you or if you have the requirement to have only a single job.