How to wake-up/sleep Azure VM's?

Using Jenkins
1

Hi, we already use Jenkins with on-premise agents and with Azure VM agents. Our jobs are typically triggered by svn commits.

Currently, the Azure VMs are up according to a schedule, regardless of whether they are actually needed for jobs. We want to look at using Jenkins to wake the VM for just the duration of the job.

Note: Our IT prefer us to use web hooks rather than service principals.

Are there any recommendations for how to do this?

I assume that every job will require stages on the Controller as well as on the Agent, to execute the web hook. Is this good practice?

How do we synchronise jobs so that one job doesn’t take down the VM while another job is running on it?

2

I would have said use Azure VM Agents plugin but as your wrote your IT doesn’t like you to use service principals that is excluded.

Currently, the Azure VMs are up according to a schedule, regardless of whether they are actually needed for jobs. We want to look at using Jenkins to wake the VM for just the duration of the job

With schedule I assume you mean a schedule in azure not directly the availability setting in Jenkins (maybe you use that as well to avoid that azure shuts down the VM while a build is running)

How many agents do you have with the same label and how many executors do the agents have? With a single executor you could use lockable resources plugin to get a resources with a label and the resource name being the agent name. After you got the lock you read the agent name, trigger the webhook then use node(agent-name) this will wait until the agent is running, execute your stuff, then afterwards you trigger shutdown of the VM via webhook before you leave the lock step
So create for each VM a resource in lockable resources section in global config

image
image3231×832 39.2 KB

The host1 should be the agent name in Jenkins not any label of that agent.

lock(label: 'label1', quantity: 1, variable: 'lockedResource') {
  resourceName = env.lockedResource // that should be "host1"
  // trigger webhook that starts the VM
  node(resourceName) {
  }
  // trigger webhook that stop VM
}

It means the VM is stopped again after each build and not reused when another is already waiting for a lock

And as mentioned this will only work when the agents have only 1 executor. It also means all jobs must adhere to this way of using the agent. If any job uses the agent without that mechanism there is the risk that a job is started after the the node step finished but before the VM is stopped.

1 Like
3

Thank you Markus, that is very helpful.

4

I would like to ask a couple more questions about how best to implement this please.

Firstly, multiple pipelines (that use the same lockable agent) will each need to know the webhook for waking/shutting down the agent. Is there a convenient way in Jenkins for storing this data (the webhook url) centrally, accessible by each pipeline?

Secondly, the wake/shutdown stages of each pipeline must run on a node with multiple executors and which is not lockable. Must a top-level agent be specified for the pipeline, and should that not be the built-in node?

5

for your first point lockable resources plugin allows to store properties for each resource. So you can just store it there in a property named HOOK and then use like this (untested)

lock(label: 'some_resource', variable: 'lockedResource') {
def  hook =  env.lockedResource0_PROP_HOOK
...
}

A top level agent is not needed if all you need to do is call a url. With the HTTP Request plugin you can do http requests without an agent I think
If you need to call a script you could just add more node steps. If possible avoid to run anything on the built-in node

lock(label: 'label1', quantity: 1, variable: 'lockedResource') {
  def resourceName = env.lockedResource // that should be "host1"
  def hook =  env.lockedResource0_PROP_HOOK
  node("some label") {
    // call script that stops agent
    sh """
      myscript $hook start 
    """
  }
  node(resourceName) {
  }
  node("some label") {
    // call script that stops agent
    sh """
      myscript $hook stop 
    """
  }
}

Finally I would put that code in a shared pipeline library so you don’t need to duplicate the code for each project.
In the end it should just be something like

withAzureVM("label1") {
  // here is the code that should be run on the azure vm
}
2 Likes