Jenkins agent time off

poddingue · July 25, 2023, 8:18am

Hello @maiductuan and welcome to this community.

I think you can use the Jenkins API to get information about offline agents and check if they lost connection to Jenkins. The Jenkins API provides various endpoints that you can use to retrieve information about agents and their status.

We could start by using the following API endpoint: http://your-jenkins-server/computer/api/json.

This endpoint will return a JSON representation of all agents, including offline agents. For each agent, you will find a boolean in the offline attribute, indicating if the agent is offline or not. There is also a second boolean named temporarilyOffline that could help you find if that was a transient issue or not.

Here’s an example using curl and jq to get the list of offline agents:

# Replace 'your-jenkins-server' with your Jenkins server URL
JENKINS_SERVER="http://your-jenkins-server"

# Get JSON representation of all agents
AGENTS_JSON=$(curl -s "${JENKINS_SERVER}/computer/api/json")

# Parse the JSON using jq and filter agents that have been offline for more than 3 months
OFFLINE_AGENTS=$(echo "${AGENTS_JSON}" | jq -r '.computer | map(select(.offline)) ')

# Print the result
echo "${OFFLINE_AGENTS}" | jq

I get something like:

[
  {
    "_class": "hudson.slaves.SlaveComputer",
    "actions": [
      {}
    ],
    "assignedLabels": [
      {
        "name": "GCP"
      },
      {
        "name": "docker"
      },
      {
        "name": "linux"
      }
    ],
    "description": "GCP, if you're lucky enough to have one",
    "displayName": "GCP",
    "executors": [
      {},
      {},
      {},
      {}
    ],
    "icon": "symbol-computer-offline",
    "iconClassName": "symbol-computer-offline",
    "idle": true,
    "jnlpAgent": true,
    "launchSupported": false,
    "loadStatistics": {
      "_class": "hudson.model.Label$1"
    },
    "manualLaunchAllowed": true,
    "monitorData": {},
    "numExecutors": 4,
    "offline": true,
    "offlineCause": null,
    "offlineCauseReason": "",
    "oneOffExecutors": [],
    "temporarilyOffline": false,
    "absoluteRemotePath": null
  }
]

Once you have retrieved the name of each offline agent (and if you know the name of the jobs it’s supposed to run), you could use this endpoint: http://your-jenkins-server/job/<job-name>/lastBuild/api/json?tree=timestamp.
But we’re missing the link between the agents and the jobs…

I have made this script that works for me.
It may not work at all for you:

github.com

gounthar/jenkins-remote-utils/blob/ccdb24313de99f482d51d33a054e2377c59a733c/sort_agents_by_usage.sh

#!/bin/bash

# Before running this script, make sure to set the following environment variables:
# JENKINS_URL: The URL of your Jenkins server (e.g., http://localhost:8080)
# USERNAME: Your Jenkins username (e.g., admin)
# API_TOKEN: Your Jenkins API token (generate one from Jenkins user settings)

# Check if the required environment variables are set
if [[ -z "$JENKINS_URL" || -z "$USERNAME" || -z "$API_TOKEN" ]]; then
  echo "Error: Please set the required environment variables before running this script." >&2
  echo "Required environment variables: JENKINS_URL, USERNAME, API_TOKEN" >&2
  exit 1
fi

TEMP_DEBUG_FILE=$(mktemp)

# Associative array to store the job builds
declare -A job_builds
# Nested associative array to store the tree-like structure
declare -A job_tree

This file has been truncated. show original

It’s not efficient, beautiful, or even well thought out, but it does exist.
It gathers all agents and all defined jobs.
For each multibranch pipeline, it then gathers all sub-pipelines.
For each pipeline, it then retrieves the last 10 jobs.
For each of these jobs, it then retrieves the agents used, and the associated timestamp.
In the end, you get a list of detected agents and their last use if any.

For example:

./sort_agents_by_usage.sh
[INFO] TEMP_DEBUG_FILE: /tmp/tmp.HIT6lAsn3B
[INFO] TEMP_DEBUG_FILE: /tmp/tmp.HIT6lAsn3B
Jenkins Job Summary

Found Agents:
Agent: Built-In
Agent: GCP
Agent: android-agent
Agent: Node
Agent: jenkins-agent
Total Agents: 5


Finding jobs
AutoUpdaterMultiBranches
FirstJob
MultiBranchAndroidPipeline
PipelineJob
PublishAppToGitHubAndGoogle

Finding sub jobs in multibranch pipelines
[MESSAGE] [process_job] Processing multibranch job: AutoUpdaterMultiBranches with path AutoUpdaterMultiBranches
[MESSAGE] [process_job] AutoUpdaterMultiBranches is a multibranch pipeline job
[MESSAGE] [process_job] PR-331 is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] PR-393 is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] PR-394 is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] PR-395 is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] aarch64 is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] dependable-pipelines is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] gitpod-ibrahim is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] main is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] qodana is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] stf is a sub job of AutoUpdaterMultiBranches
[MESSAGE] [process_job] Sub-jobs for AutoUpdaterMultiBranches: PR-331
[MESSAGE] [process_job] Processing multibranch job: MultiBranchAndroidPipeline with path MultiBranchAndroidPipeline
[MESSAGE] [process_job] MultiBranchAndroidPipeline is a multibranch pipeline job
[MESSAGE] [process_job] PR-331 is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] PR-395 is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] aarch64 is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] dependable-pipelines is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] does-docker-image-alerady-exists is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] emulator is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] gitpod is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] gitpod-ibrahim is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] main is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] new-dockerfile-true is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] qodana is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] release-drafter is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] stf is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] vagrant is a sub job of MultiBranchAndroidPipeline
[MESSAGE] [process_job] Sub-jobs for MultiBranchAndroidPipeline: PR-331
[MESSAGE] [process_job] Processing freestyle job: FirstJob with path FirstJob
[MESSAGE] [process_job] Processing pipeline job: PipelineJob with path PipelineJob
[MESSAGE] [process_job] Processing pipeline job: PublishAppToGitHubAndGoogle with path PublishAppToGitHubAndGoogle

Displaying raw associative arrays
[MESSAGE] Associative array "job_builds" contents:
[MESSAGE] Key: MultiBranchAndroidPipeline/stf, Value: 18 17 16 15 14 13 12 11 10 9 8 7 5 4 3 2 1
[MESSAGE] Key: MultiBranchAndroidPipeline/gitpod, Value: 2
[MESSAGE] Key: MultiBranchAndroidPipeline/PR-331, Value: 14 13 12 11 10 9 8 7 6 5 4 3 2 1
[MESSAGE] Key: MultiBranchAndroidPipeline/vagrant, Value: 1
[MESSAGE] Key: PublishAppToGitHubAndGoogle, Value: 1
[MESSAGE] Key: AutoUpdaterMultiBranches/dependable-pipelines, Value: 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
[MESSAGE] Key: AutoUpdaterMultiBranches/gitpod-ibrahim, Value: 5 4 3 2 1
[MESSAGE] Key: AutoUpdaterMultiBranches/PR-393, Value: 1
[MESSAGE] Key: AutoUpdaterMultiBranches/PR-394, Value: 1
[MESSAGE] Key: AutoUpdaterMultiBranches/PR-395, Value: 1
[MESSAGE] Key: MultiBranchAndroidPipeline/dependable-pipelines, Value:
[MESSAGE] Key: MultiBranchAndroidPipeline/emulator, Value:
[MESSAGE] Key: MultiBranchAndroidPipeline/gitpod-ibrahim, Value: 3 2
[MESSAGE] Key: AutoUpdaterMultiBranches/stf, Value: 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
[MESSAGE] Key: MultiBranchAndroidPipeline/PR-395, Value: 1
[MESSAGE] Key: MultiBranchAndroidPipeline/new-dockerfile-true, Value:
[MESSAGE] Key: MultiBranchAndroidPipeline/does-docker-image-alerady-exists, Value:
[MESSAGE] Key: PipelineJob, Value: 1
[MESSAGE] Key: MultiBranchAndroidPipeline/qodana, Value: 3 2
[MESSAGE] Key: MultiBranchAndroidPipeline/aarch64, Value: 3 2
[MESSAGE] Key: MultiBranchAndroidPipeline/release-drafter, Value: 1
[MESSAGE] Key: AutoUpdaterMultiBranches/main, Value: 49 48 47 46 45 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
[MESSAGE] Key: AutoUpdaterMultiBranches/aarch64, Value: 3 2 1
[MESSAGE] Key: AutoUpdaterMultiBranches/PR-331, Value: 9 8 7 6 5 4 3 2 1
[MESSAGE] Key: MultiBranchAndroidPipeline/main, Value: 28 27 26 25 24 23 22 21 19 18 17 16 15 14 13 12 11 10 9
[MESSAGE] Key: AutoUpdaterMultiBranches/qodana, Value: 5 4 3 2 1
[MESSAGE] Key: FirstJob, Value: 11 10 9 8 7 6 5 4 3 2 1
[MESSAGE] Associative array "job_tree" contents:
[MESSAGE] Key: PublishAppToGitHubAndGoogle, Value:
[MESSAGE] Key: MultiBranchAndroidPipeline, Value: PR-331 PR-395 aarch64 dependable-pipelines does-docker-image-alerady-exists emulator gitpod gitpod-ibrahim main new-dockerfile-true qodana release-drafter stf vagrant
[MESSAGE] Key: PipelineJob, Value:
[MESSAGE] Key: AutoUpdaterMultiBranches, Value: PR-331 PR-393 PR-394 PR-395 aarch64 dependable-pipelines gitpod-ibrahim main qodana stf
[MESSAGE] Key: FirstJob, Value:

Agent Summary:
Agent: Built-In
Usage Count: 0
Last Used Timestamp: 0

Agent: GCP
Usage Count: 0
Last Used Timestamp: 0

Agent: android-agent
Usage Count: 267
Last Used Timestamp: 1690268568005

Agent: Node
Usage Count: 0
Last Used Timestamp: 0

Agent: jenkins-agent
Usage Count: 170
Last Used Timestamp: 1690268568005

Total Agents: 5

Sorted Agents:
Agent: android-agent
Usage Count: 267
Last Used Timestamp: 1690268568005

Agent: jenkins-agent
Usage Count: 170
Last Used Timestamp: 1690268568005

Agent: Built-In
Usage Count: 0
Last Used Timestamp: 0

Agent: GCP
Usage Count: 0
Last Used Timestamp: 0

Agent: Node
Usage Count: 0
Last Used Timestamp: 0

[INFO] TEMP_DEBUG_FILE: /tmp/tmp.HIT6lAsn3B

Topic		Replies	Views
How can I retrieve the status of the agent? Community	2	740	November 6, 2023
How to get the status of slave machine Using Jenkins question	2	665	September 18, 2023
Agents disconnecting after a period of time Ask a question	4	897	December 7, 2023
Need Rest API for finding last 5 jobs run on particular Agent Ask a question	1	24	October 9, 2024
Question for jenkins agent response time Using Jenkins question	2	1194	May 8, 2022

Jenkins agent time off

Related topics