I upgraded Jenkins from 2.204.5 to 2.440.2 by replacing the jenkins.war file. After restarting Jenkins, everything looked good for about a month. But, yesterday, I was getting exceptions while using the console. I later found that the config.xml (/var/lib/jenkins) was over 150MB and it contained console output from the builds. How do I fix this?
Restore that file from a backup. Check the file system for corruption and repair the file system level corruption with operating system utilities. Configure Jenkins to have 0 executors on the controller so that you reduce the risk of a misconfigured job doing damage
I restored the config.xml from a backup y’day, but I see that a lot of build information is being written into that file again. Are there additional changes to make besides swapping out the jenkins.war for the upgrade?
You’ll need to find whatever is writing that data to the file, whether a Job run by Jenkins on the controller (disable all executors on the controller) or by some configuration error of Jenkins itself, and fix that problem.
If you have trouble figuring out what job is writing there, try making jenkins.xml read only and the you’d probably find the error in the jenkins.log or the job’s logs.
1 Like
I’m running Jenkins using the built-in node (continuing with the legacy model used since 5 yrs back). So, disabling the executors would basically run none of the jobs, right?
Yes, I tried that by setting config.xml to read-only, After I restarted the Jenkins service, I noticed that the file permissions changed to -rw-r–r–
1 Like
That’s right, you have to set up an agent outside of the controller.
It could be on the same machine, though.
Ok, let me try that on Linux. I’ve checked all the configurations and still can’t figure out why the build information is being written to config.xml
1 Like
So, I monitored this on 4 servers since yesterday. I restored the /var/lib/jenkins/config.xml (from an earlier backup) and set up a simple loop to check the file size (on all 4 servers). Interestingly, the file got corrupted 1 minute after midnight on all 4 servers. I have restored the config.xml again and am monitoring when the corruption happens again.
1 Like
What does your crontab say?
Would you have any logs about what happens at midnight?
Ok, the issue was with a misconfigured cron job, that was added recently. It’s all good now.
2 Likes
Well done! 
Thanks a lot for your feedback. 