Jenkins deletes the workspace even when not checked

Using Jenkins
1

I have a site that I’m deploying via Jenkins and due to the large number of images that’s part of the site, I’ve not checked the ‘Delete Workspace’ checkbox. However upon review, it appears even unchecked, the workspace is deleted. It takes about 20 minutes to rsync the image files into the workspace even though there are only a few new images to be added.

Is there something I’m not understanding both in the delete workspace space or a better way of integrating binary files into a project which is then sync’d with the remote live site.

Thanks!

Carl

2

Can you go into how your testing your assumptions/came to your conclusion? It actually takes lot of effort to delete workspaces, but it may be spinning up a new one.

3

Sure. I logged into the Jenkins worker and ran a while true; do ls; sleep 3; done and ran the Jenkins job. I watched as the files and directories were deleted over the course of a few refreshes, then the directory structure and files were recreated.

I did enable ‘Delete Workspace’ and watch and all the files were deleted immediately, it didn’t take any time unlike when it was unchecked.

Someone on reddit suggested deleting the plugin which might solve my issue but certainly not the general issue of the delete and recreate.

For a majority of my projects, a delete and recreate is a non issue. But this is a pretty large photo site and it’s taking 20 minutes to run.

Carl

4

do you remember the path relative to $JENKINS_HOME that was deleted? if it ended in @tmp or @2 it probably won’t last long.

Are you talking about the files in $JOB/workspace or something else?

5

I was in /var/lib/jenkins/workspaces/photos. Not any of the @ directories.

6

Okay, I’ve deleted the Workspace Cleanup plugin however a rebuild of the project as noted in the output below has deleted the workspace directory entirely and then started adding it back in again, so again, a 20 minute process to build the directory structure and incorporate the binaries before syncing the directory with the local site.

==============================================
1957      20040725  20050409  20060127  20061026  20070609  20070818  20090420  20120520     admin          functions     minifunc.php  tag.cloud.php
1958      20040727  20050514  20060318  20061226  20070610  20071207  20090621  2013         android.css    gallery.php   mininew.php   tag.toggle.php
1959      20040814  20050526  20060506  20061228  20070623  20071211  20090625  2014         apple.css      gifshow.php   mobile.php    tag.view.php
1960      20040820  20050611  20060527  20061230  20070630  2008      20090720  2015         confooter.php  header.php    planning      topmenu.end.php
2000      20040912  20050621  20060610  2007      20070701  20080302  20090808  2016         css            imgs          pngshow.php   topmenu.start.php
2001      20040925  20050811  20060714  20070304  20070707  20080410  2010      2017         docs           index.php     robots.txt    weight
2002      20041106  20050820  20060727  20070311  20070714  20080416  20100417  2018         dominator      jpegshow.php  routes        years.php
20021215  20041203  20050904  20060730  20070317  20070715  20080608  2011      2019         favicon.gif    jpgshow.php   show.php
2003      2005      20051001  20060819  20070525  20070722  20080620  20110406  2020         favicon.ico    login         site.css
20030808  20050123  20051015  20060924  20070526  20070729  20080920  20110725  2021         footer.php     maintenance   states
2004      20050226  2006      20061018  20070527  20070811  2009      2012      account.php  function.php   menu.css      stuff
==============================================
==============================================
20051001  20070729  20100417     admin          footer.php    header.php    maintenance   planning       tag.toggle.php     years.php
20060610  20080620  20110406     android.css    function.php  index.php     menu.css      pngshow.php    tag.view.php
20060924  20080920  20110725     apple.css      functions     jpegshow.php  minifunc.php  show.php       topmenu.end.php
20070317  20090720  20120520     confooter.php  gallery.php   jpgshow.php   mininew.php   site.css       topmenu.start.php
20070722  20090808  account.php  css            gifshow.php   login         mobile.php    tag.cloud.php  weight
==============================================
7

So, out of luck then I guess. It just takes 20 minutes to update a site because Jenkins for some unknown reason deletes the workspace. I wonder why there’s a ‘Workspace Cleanup’ plugin then.

8

While files are being deleted, follow instructions from Obtaining a thread dump and post the output somewhere.

1 Like
9

Thank you. Will follow the instructions and make it available.

10

I’ve captured the threaddump output and stored it here:

https://schelin.s3.amazonaws.com/jenkins.threaddump

Carl

11

At the time the thread dump was taken, a single build is executing and it’s running some sort of script (shell/Batch/…). If this really is when things get deleted, review your job configuration and build script. To narrow it down further, we’d need a thread dump from the agent process.

12

After the git repo is pulled from gitlab, I run a few shell commands to merge in the items that aren’t in the git repo, images being the main thing.

The process would be:

  1. Jenkins sees a change in gitlab/the job is run manually and the repo is put into the WORKSPACE directory.
  2. The shell command merges in the /css directory structure using rsync.
  3. The shell command merges in the images using rsync.
  4. The .git directory and a few files are removed.
  5. The final WORKSPACE is rsync’d with the remote websites.

The deletion appears to occur before step 1 as all the files from the git repo are in the final WORKSPACE.

Should I have two browser tabs open, one with the threaddump open and then on the first one, kick off the job?

And I do appreciate the help. Thanks!

13

I suspect that step 4 is causing the problem in combination with JENKINS-22795

By performing step 4, you are assuming that the git plugin will repopulate the .git directory without harming the rest of the repository. There were cases in early use of the git plugin where the failure to empty a directory before a fresh clone would cause problems that were very difficult to diagnose. By removing the .git directory, you’ve caused the git plugin to believe that it must perform a fresh clone. A fresh clone in the git plugin assumes that it has full control of the workspace and empties the workspace so that the initial git clone has the best chance of success.

By removing the .git directory, you’re also assuring that the next run will need to repopulate that directory. Since it contains a complete copy of the history of the repository, that should generally be avoided.

Don’t remove the .git directory. Configure your rsync command to ignore the .git directory and its contents.

14

That’s an idea. I’m using a similar process with gitlab-ci and it deletes the working directory again as well. One of the folks on my team recommended an entirely different directory with the artifacts as well.

I’ve used rsync with exclude before for other purposes. I’ll give that a try, thanks!

15

That appears to be the solution. I added the exclude to the rsync and didn’t delete the .git directory and it doesn’t delete the directory any more. Oddly a second exclude or a {} exclude or even an exclude-from doesn’t exclude the .gitlab-ci.yml file from copied. I’ve used exclude-from to not copy several files over in other situations without issue so this is a bit odd. I’ll puzzle it out though.

Thanks @MarkEWaite