Hello Jenkins Community,
Hello and good morning.
I am looking for advice on the following issue, please.
This affects both our production and test environments.
This started happening at the same time in both.
Google searches have provided some workaround ideas.
Has anybody found a solution for this?
Please see below, 1-4
My name is John Dove.
I am a Jenkins administrator for an ITS department, focused on Jenkins/Maven/Java development. I have been running the same Jenkins system for several years now (4 years) with NO major problems. Our Jenkins is older (2.150.1) but very (very) stable and quite an excellent product.
We have a Jenkins production system.
We have a Jenkins Test system.
Both are being affected.
Starting now, in OCTOBER 2023, we have a mysterious problem.
This problem affects both our Production and Test systems.
This problem just started affecting both systems at the SAME time.
Jenkins and SVN software has NOT changed in 4 years.
A very stable environment.
JENKINS ERROR - when doing SVN checkout:
svn: E175002: REPORT request failed on '/ij/!svn/vcc/default'
SVN ERROR - at the same time
[dav:error]  ] Provider encountered an error while streaming a REPORT response. [500, #0]
[dav:error] A failure occurred while driving the update report editor [500, #104]
[dav:error]  Error writing base64 data: Connection reset by peer [500, #104]
This problem occurs INTERMITTENTLY and NOT all the time.
Some projects finish their SVN checkouts.
Some fail with the above errors.
When this does happen, both JENKINS and SVN always have the same errors as above.
This problem appears to be a known issue.
Referenced back in year 2008.
It was replied to by: Kohsuke Kawaguchi
ISSUE: Loading... “svn: REPORT request failed on ‘/svn/!svn/vcc/default’”
And yes we tried the recommendation to use “-Dhudson.spool-svn=true” – but that did not work.
I am still actively researching this issue on my side. I will keep this Jenkins Forum thread updated with any progress that I make (good or bad). Hoping for advice.
Not sure if that option is still relevant but it looks like the subversion plugin provides a couple of properties to change its connecting behavior. Worth looking through the plugin source to see if any of the options not mentioned yet would help.
“Connection reset by peer” is a generic SSH error so might be worth tweaking those options. 4 years stability is a long time. I imagine Jenkins usage and source code have grown. As systems grow they can lose stability or face underlying issues. I am experiencing that myself recently but in a different setup.
Did something in your network change? Are you absolutely sure that nobody updated something on the Jenkins side? Did the svn server get updated? If what you say is true, that nothing in your Jenkins side changed, then you need to look outside that area.
If your SVN plugin is at least 4 years old, it’s possible that the SVNKit version it is using can’t handle something in the network. There may be possible bugs in that library that the SVN plugin uses to interact with SVN.
You should probably change SVN version in settings. I had various problems with SVN plugin until I modified this. Don’t know why but the plugin is using a very old repository version by default (don’t remember if it was 1.6 or even 1.4). I mean I know it is a compatibility thing, but I don’t why is it that low by default.
You should go to manage/configure and probably change to 1.8.
Note that you will probably need to remove workspace for any and all SVN based builds after changing this. IIRC the plugin will not remove existing checkouts.
I’ve found my notes. The default is 1.4. You can check in .svn/format file (its a text file, but do not modify it manually!). You should have format version 12 (1.7+) to support more modern features of SVN.
I did lots of testing. Definitely appears that my Jenkins SVN checkout problem is environmental of some kind. I installed my Jenkins system software (same version / same plugins / etc.) to a different machine environment, and SVN checkouts worked. Machine-environment was the only difference.
I will keep this forum updated.
More testing awaits.
Just to keep the community updated on this. I am making some good progress. I followed the advice from this JENKINS issue from past years (year 2008; JENKINS-1260, which was mentioned above) specifically for using the below -D Java JVM parameter. And it seems to be working. Meaning, that my recent SVN CHECKOUT tests have all been 100% successful so far.
What does the above -D setting actually (literally) do technically at runtime?
I saw its source code on GIT HUB, which is just an explicit constructor for the DAV Repo Factory. DAVRepositoryFactory.setup(new DefaultHTTPConnectionFactory(null,true,null));
Here is the source code.
But under-the-hood, does anyone know what this spooling is doing?
Seems to work. Why?
-Dhudson.spool-svn=true setting is working so far in isolated JENKINSMASTER hosted in TOMCAT. Next step is trying to instate within distributed agent environment of 4 SLAVES. Will keep this forum update as I move along.
This applies to my TEST architecture only.
I applied the -Dhudson.spool-svn=true setting to my distributed, NON-production Jenkins architecture, consisting of 1 JENKINSMaster and 4 SLAVES. They are all Microsoft windows machines, such that all 4 SLAVES run as Windows Services on 4 independent machines. JENKINSMaster is an APACHE Web Server reverse proxy to (Tomcat + Jenkins) and Sonar Qube.
SVN CHECKOUTS are working on all SLAVES now.
Again, in NON-production.
(Production is next…)
The “technical fix” was to simply add the above -Dhudson.spool-svn=true JVM property to to ALL SLAVES’ Jenkins.XML file. That’s it. And it worked.