Suggestion to run particular build only once in each agent

Using Jenkins Ask a question
1

Hi there,

We are using the Amazon EC2 instance as jenkins agent and the multibranch pipeline build is trigger for any changes detected in SCM. Now, one particular pipeline build is consume huge heap memory and we want to run this particular build in one agent only, but other build which is consume less heap size can run in the same agent.

So, here is the situation. for example:- we have A, B, C pipelines and C is assume more heap memory

  1. A/B/C first build is trigger and it can initiate/spin up ec2 agent and start build in there.
  2. But when the second build is trigger from A/B/C pipelines, A/B can go to any agent but C will check the existing running ec2 agent and run the build if the similar build is not running or else spin up new agent.

Looking for suggestion, how to configure the C pipeline build runs once in each agent.

Thanks

2

Please remember to use agent instead of slave - On Jenkins Terminology Updates - I’ve updated your post.

1 Like
3

In your ec2 agent configuration, make sure it has a distinct label like “big” and then under the Usage parameter select Use this node as much as possible.

for pipeline A/B use:

node() {
}

and for C:

node('big') {
}

A/B will run on any agent available, while C will only run on the agent labeled “big”. The Pipeline: Nodes and Processes step actually supports lots of options for controlling which agent your build is run on.

4

If I’m understanding you correctly, you want a node running job C to be able to pick up a second job of type A or B, but to run another job C, you need to spin up a fresh EC2 machine for it?

We’ve got something similar set up at my company using the Throttle Concurrents plugin (docs), but it uses statically-allocated executors which are always present, not dynamic executors spun up through a cloud plugin. However, give it a try; it may end up working just fine.

5

Thanks we are using the tag to route the different pipelines build to different agent based on their build requirement.

6

Yeah similar, if the second job of C is trigger then it will look at the existing running agent and if no C job is running then trigger the build on existing agent/s otherwise spinup new agent

7

@mtughan , How it work for you in static agents.
I used this pipeline syntax based on their document

agent { label 'docker' }
  options {
      throttleJobProperty(
          categories: ['throttle-test'],
          limitOneJobWithMatchingParams: false,
          throttleEnabled: true,
          throttleOption: 'category'
      )
  }

and then declare the “throttle-test” policy for concurrent builds in Global configuration
image

But still build is running to the nodes which is already running the a existing build.

Wondering, what am I missing here!?

8

Even I tried Throttle in project alone but still can’t restrict the build to run 1 in each node :frowning:

options {
        throttleJobProperty(
        categories: [],
        limitOneJobWithMatchingParams: false,
        maxConcurrentPerNode: 1,
        maxConcurrentTotal: 4,
        paramsToUseForLimit: '',
        throttleEnabled: true,
        throttleOption: 'project'
      )
9

Hi Sukanya,

Everything you’re doing here appears to be correct, so I’m unsure why it’s not working properly. The only thing I can think of is that we’re using freestyle jobs instead of pipeline jobs, so maybe it’s broken for pipeline jobs. In particular, it may be related to JENKINS-63120, so maybe it won’t currently work for your use case.