Jenkins SVN Checkout Issue svn: E175002

jodove · October 20, 2023, 3:58pm

Hello Jenkins Community,
Hello and good morning.
I am looking for advice on the following issue, please.
This affects both our production and test environments.
This started happening at the same time in both.
Google searches have provided some workaround ideas.
Has anybody found a solution for this?
Please see below, 1-4
Thanks,
John Dove

1
My name is John Dove.
I am a Jenkins administrator for an ITS department, focused on Jenkins/Maven/Java development. I have been running the same Jenkins system for several years now (4 years) with NO major problems. Our Jenkins is older (2.150.1) but very (very) stable and quite an excellent product.
We have a Jenkins production system.
We have a Jenkins Test system.
Both are being affected.

2
Starting now, in OCTOBER 2023, we have a mysterious problem.
This problem affects both our Production and Test systems.
This problem just started affecting both systems at the SAME time.
Jenkins and SVN software has NOT changed in 4 years.
A very stable environment.

JENKINS ERROR - when doing SVN checkout:
  svn: E175002: REPORT request failed on '/ij/!svn/vcc/default'

SVN ERROR - at the same time
[dav:error] [] ] Provider encountered an error while streaming a REPORT response.  [500, #0]
[dav:error] []A failure occurred while driving the update report editor  [500, #104]
[dav:error] [] Error writing base64 data: Connection reset by peer  [500, #104]

This problem occurs INTERMITTENTLY and NOT all the time.
Some projects finish their SVN checkouts.
Some fail with the above errors.
When this does happen, both JENKINS and SVN always have the same errors as above.

3
This problem appears to be a known issue.
Referenced back in year 2008.
It was replied to by: Kohsuke Kawaguchi
ISSUE: Loading... “svn: REPORT request failed on ‘/svn/!svn/vcc/default’”
And yes we tried the recommendation to use “-Dhudson.spool-svn=true” – but that did not work.

4
I am still actively researching this issue on my side. I will keep this Jenkins Forum thread updated with any progress that I make (good or bad). Hoping for advice.
Thanks.
~ John

jodove · October 20, 2023, 5:54pm

I removed the picture I posted with this.
Updated so just text.

samrocketman · October 24, 2023, 12:10am

Here is that hudson.spool-svn option in the subversion plugin. As well as original commit in jenkins core before plugin breakout.

Not sure if that option is still relevant but it looks like the subversion plugin provides a couple of properties to change its connecting behavior. Worth looking through the plugin source to see if any of the options not mentioned yet would help.

“Connection reset by peer” is a generic SSH error so might be worth tweaking those options. 4 years stability is a long time. I imagine Jenkins usage and source code have grown. As systems grow they can lose stability or face underlying issues. I am experiencing that myself recently but in a different setup.

jodove · October 24, 2023, 1:24pm

Sam,

Thanks for the links I will look at them.

Is there anything else you can think of that could help us?

Where could the problem possibly be?

Since the software has been used for 4 years with no problems (at all, ever) and now this with the errors being thrown on SVN checkout, can you think of any possible things?

slide_o_mix · October 24, 2023, 2:13pm

Did something in your network change? Are you absolutely sure that nobody updated something on the Jenkins side? Did the svn server get updated? If what you say is true, that nothing in your Jenkins side changed, then you need to look outside that area.

jodove · October 24, 2023, 4:00pm

Hi Alex

Thank you for the reply.

My replies as follows.

Below, 1-5.

~ John

1

Did something in your network change?

Yes.

Recently (around THIS timeframe October 2023) there have been some changes to the network by our ITS networking team.

It is indeed possible those network changes effected the “network-ecosystem” where our Jenkins lives.

2

Are you absolutely sure that nobody updated something on the Jenkins side?

Yes.

I am the primary person responsible for updating our Jenkins software (and any software related to Jenkins) on the concerned machines.

I have upgraded nothing.

ITS does normal patching of the operating systems. Outside of my team.

3

Did the svn server get updated?

No.

Our SVN servers (TEST and PROD) have NOT been upgraded in years.

ITS does normal patching of the operating systems. Outside of my team.

4

If what you say is true, that nothing in your Jenkins side changed, then you need to look outside that area.

We thought the same.

We did Wiresharking inspection with ITS networking teams already – watching when Jenkins agent sends its “SVN checkout” requests to SVN servers.

Wireshark revealed some “TCP ZEROWINDOW” packets - but just a slight few – which hints at a performance issue concerning data I/O where one side in the I/O is suffering (not ready for its data, etc.).

Was mentioned could be a possible concern.

5

Here is a really off-the-wall thought – more common sense:

Could a beneficial network change (allowing better & faster network IO performance) actually cause the Jenkins SVN PLUGIN to choke and fail in handling data?

Make no sense since we are dealing with computers, but in metaphor-like-thinking, trying to force too much WATER into a SMALLER pipe?

Could that even theoretically come into play?

Just a thought.

jodove · October 24, 2023, 4:14pm

I am adding this diagram for everyone else’s understanding to illustrate this problem.
More academic this way

slide_o_mix · October 24, 2023, 4:20pm

If your SVN plugin is at least 4 years old, it’s possible that the SVNKit version it is using can’t handle something in the network. There may be possible bugs in that library that the SVN plugin uses to interact with SVN.

In addition, this post (tortoisesvn - Error REPORT request failed on ../../../!svn/vcc/default - Stack Overflow) seems to hint that it could possibly be related to a few different things (two SVN versions accessing the same working directory, file corruption in the repo, etc.). It might be helpful to take a look there and see if any of the “fixes” discussed help resolve your issue.

jodove · October 24, 2023, 4:48pm

Alex,

Thanks. Good ideas.

I was thinking the SVNKIT could be it too !

Thanks for all the input.

By the way.

I did testing with Jenkins 2.346.3 + SVN PLUGIN 2.17.1 – which are more current versions of Jenkins + SVN PLUGIN.

Same problem occurred with them too.

I will keep testing…

Thanks again for all the input.

Nux · October 25, 2023, 6:08pm

You should probably change SVN version in settings. I had various problems with SVN plugin until I modified this. Don’t know why but the plugin is using a very old repository version by default (don’t remember if it was 1.6 or even 1.4). I mean I know it is a compatibility thing, but I don’t why is it that low by default.

You should go to manage/configure and probably change to 1.8.
obraz

Note that you will probably need to remove workspace for any and all SVN based builds after changing this. IIRC the plugin will not remove existing checkouts.

Nux · October 25, 2023, 6:20pm

I’ve found my notes. The default is 1.4. You can check in .svn/format file (its a text file, but do not modify it manually!). You should have format version 12 (1.7+) to support more modern features of SVN.

jodove · October 25, 2023, 7:11pm

Thanks for the good ideas

: )

jodove · October 25, 2023, 7:16pm

Quick update

I did lots of testing. Definitely appears that my Jenkins SVN checkout problem is environmental of some kind. I installed my Jenkins system software (same version / same plugins / etc.) to a different machine environment, and SVN checkouts worked. Machine-environment was the only difference.

I will keep this forum updated.
More testing awaits.
Interesting.

~ John

jodove · October 27, 2023, 7:53pm

Quick update

Hello All,
Just to keep the community updated on this. I am making some good progress. I followed the advice from this JENKINS issue from past years (year 2008; JENKINS-1260, which was mentioned above) specifically for using the below -D Java JVM parameter. And it seems to be working. Meaning, that my recent SVN CHECKOUT tests have all been 100% successful so far.

     -Dhudson.spool-svn=true

Question:
What does the above -D setting actually (literally) do technically at runtime?
I saw its source code on GIT HUB, which is just an explicit constructor for the DAV Repo Factory. DAVRepositoryFactory.setup(new DefaultHTTPConnectionFactory(null,true,null));

Here is the source code.

github.com

jenkinsci/subversion-plugin/blob/ff80157fb7312817f237f26463ae4376337d9816/src/main/java/hudson/scm/SubversionSCM.java#L2689


      
              } else {
                  listener.getLogger().println(Messages.SubversionSCM_disableProject_unsupported());
              }
          }
          
          static final Pattern URL_PATTERN = Pattern.compile("(https?|svn(\\+[a-z0-9]+)?|file)://.+");
          
          private static final long serialVersionUID = 1L;
          
          public static void init(int workspaceFormat) {
                  if(Boolean.getBoolean("hudson.spool-svn"))
                      DAVRepositoryFactory.setup(new DefaultHTTPConnectionFactory(null,true,null));
                  else
                      DAVRepositoryFactory.setup();   // http, https
                  SVNRepositoryFactoryImpl.setup();   // svn, svn+xxx
                  FSRepositoryFactory.setup();    // file
          
                  // disable the connection pooling, which causes problems like
                  // http://www.nabble.com/SSH-connection-problems-p12028339.html
                  if(System.getProperty("svnkit.ssh2.persistent")==null)
                      System.setProperty("svnkit.ssh2.persistent","false");

But under-the-hood, does anyone know what this spooling is doing?
Seems to work. Why?

jodove · October 30, 2023, 9:51pm

Quick Update

-Dhudson.spool-svn=true setting is working so far in isolated JENKINSMASTER hosted in TOMCAT. Next step is trying to instate within distributed agent environment of 4 SLAVES. Will keep this forum update as I move along.

John

jodove · November 1, 2023, 6:07pm

Quick Update
Good news.
This applies to my TEST architecture only.
Below 1-3.
~ John

1
I applied the -Dhudson.spool-svn=true setting to my distributed, NON-production Jenkins architecture, consisting of 1 JENKINSMaster and 4 SLAVES. They are all Microsoft windows machines, such that all 4 SLAVES run as Windows Services on 4 independent machines. JENKINSMaster is an APACHE Web Server reverse proxy to (Tomcat + Jenkins) and Sonar Qube.

2
SVN CHECKOUTS are working on all SLAVES now.
Again, in NON-production.
(Production is next…)

3
The “technical fix” was to simply add the above -Dhudson.spool-svn=true JVM property to to ALL SLAVES’ Jenkins.XML file. That’s it. And it worked.

slide_o_mix · November 1, 2023, 6:19pm

Just as an FYI, the terms SLAVE and MASTER have been deprecated in Jenkins for some time. Please use agent and controller respectively as the current terms.

jodove · November 1, 2023, 6:46pm

No problem. Understood.

I saw it kept trying to change that.

Assumed maybe a text-widget problem on the site…

Got it.

jodove · November 2, 2023, 5:14pm

Update

Great news.

The above fix worked in my distributed Jenkins production environment, too.

I will write another REPLY subsequent to this, which shows the complete solution.

More soon.

~ John

jodove · November 2, 2023, 5:29pm

Hello Everyone,
Here is my complete solution that I used for fixing the SVN CHECKOUT errors (E175002). Hopefully this will help other people as well.
Please see below, 1-3.
~ John

1
On EACH Jenkins agent machine the fix was to simply modify the Jenkins AGENT jenkins.XML file like so:

PATH
D:\BUILD_SYSTEM\jenkins_agent\	

FILE
jenkins-slave.xml

TEXT CHANGE added the spooling property here
<arguments>-Dhudson.spool-svn=true  -Xrs  -jar "%BASE%\slave.jar" ...

2
Here is a screen shot:

3
How I verified the above approach:

3.1
First to Jenkins vendor - from Jenkins’ creator Kohsuke Kawaguchi, posted year 2008. This showed the origin of the property: -Dhudson.spool-svn=true
https://issues.jenkins.io/browse/JENKINS-1260

3.2
Then to TMATE vendor – whose API was listed in the stack trace of the SVN errors. So I knew their API was involved.
https://issues.tmatesoft.com/issue/SGT-783

3.3
Then to SVN KIT, which is used internally within the Jenkins Subversion plugin itself. This shows the explicit constructor “DefaultHTTPConnectionFactory(null,true,null)” That was crucial to discover. Since Kohsuke Kawaguchi used it too on GIT HUB below.
https://svnkit-users.svnkit.narkive.com/wxBj9Sb1/enable-http-spooling-for-svnclientmanager

3.4
Then to GIT HUB showing the code Kohsuke Kawaguchi implemented (for JENKINS-1260 above)

github.com

jenkinsci/subversion-plugin/blob/ff80157fb7312817f237f26463ae4376337d9816/src/main/java/hudson/scm/SubversionSCM.java

/*
 * The MIT License
 * 
 * Copyright (c) 2004-2012, Sun Microsystems, Inc., Kohsuke Kawaguchi, Fulvio Cavarretta,
 * Jean-Baptiste Quenot, Luca Domenico Milanesio, Renaud Bruyeron, Stephen Connolly,
 * Tom Huybrechts, Yahoo! Inc., Manufacture Francaise des Pneumatiques Michelin,
 * Romain Seguy, OHTAKE Tomohiro
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,

This file has been truncated. show original

Scroll to line: ~2689 hudson.spool-svn / new DefaultHTTPConnectionFactory(null,true,null)
Which is the explicit constructor from SVN KIT above.
So I knew this must be it.

3.5
Then back to Jenkins on how to set the property on the agent.
https://wiki.jenkins.io/display/JENKINS/Installing+Jenkins+as+a+Windows+service
SEARCH FOR: The JVM launch parameters of these Windows services are controlled by an XML file jenkins.xml and jenkins-agent.xml respectively.
That’s how I knew where to place that value.

That’s it.
~ John

Topic		Replies	Views
Unable to checkout code in the jenkins from SVN Using Jenkins	4	899	February 24, 2023
org.tmatesoft.svn.core.SVNException: svn: E175002: connection refused by the server Ask a question	4	1780	March 2, 2023
Connection refused by server in jenkins when checking out code from SVN Repository Using Jenkins	1	1671	March 9, 2023
SVN Error while checkout \| org.tmatesoft.svn.core.SVNException: svn: E175002: Processing REPORT request response failed: XML document structures must start and end within the same entity. (/scm/svn/SCM_Tools/!svn/vcc/default) svn: E175002: REPORT request Ask a question	2	1190	April 27, 2023
Freestyle: SVN checkout failing for overlapping jobs Using Jenkins	3	446	September 18, 2023

Jenkins SVN Checkout Issue svn: E175002

Related Topics