Backup Cron in K8s: QuietDown Reason and Waiting for Builds to Finish

Using Jenkins
,
1

Hey,

I have Jenkins running in Kubernetes and I’m currently trying to create a backup job for it.

To do that I create a Kubernetes CronJob with the following script:

JENKINS_USERNAME=$(cat /run/secrets/jenkins-api-token/username)
JENKINS_API_TOKEN=$(cat /run/secrets/jenkins-api-token/api-token)

# put Jenkins in shutdown mode to prevent new jobs from being scheduled
kubectl -n jenkins exec jenkins-0 -c jenkins -- /bin/sh -c \
  "curl -XPOST -u $JENKINS_USERNAME:$JENKINS_API_TOKEN http://localhost:8080/quietDown"

# TODO wait for running jobs to finish

# create new backup from /var/jenkins_home/jobs
kubectl -n jenkins exec jenkins-0 -c jenkins -- /bin/sh -c \
  'tar -cvf - \
    --exclude="*/builds/*/libs/*" \
    --exclude="*/builds/*/jacoco/*" \
    --exclude="*/builds/*/simulations/*" \
    --directory=/var/jenkins_home/jobs . | gzip -9' \
> /backups/jenkins-home-jobs-only-$(date +"%Y-%m-%d-%H-%M").tar.gz

# cancel shutdown mode after successful backup
kubectl -n jenkins exec jenkins-0 -c jenkins -- /bin/sh -c \
  "curl -XPOST -u $JENKINS_USERNAME:$JENKINS_API_TOKEN http://localhost:8080/cancelQuietDown"

This works as far as it puts Jenkins in shutdown mode and then creates an archive from my jobs directory on a persistent volume mounted at /backup. (I mount a secret to the container with a user and api-token to use).

Now I have two issues: The first is, that I cannot figure out how to add a reason to the shutdown mode. I would like to add the info, that a backup is created and therefore no new builds are scheduled.

The second issue is, that I do not wait for all running builds to complete yet. is there an API to get a list of running builds? Or can I somehow directly “ask” Jenkins if there are still builds running?

Some time ago I used ThinBackup (ThinBackup) for this. And if I remember correctly I had the issue, that some builds were stuck once the controller was put in shutdown mode and the backup never actually started. Is this still an issue?

Thanks in advance!