Reverse Proxy Test does not handle URL correctly

Hello,

I am using Jenkins 2.319.1 with nginx as reverse proxy. There is a message in Jenkins that “it appears that your reverse proxy set up is broken”. I researched this problem quite a bit, and followed the usual steps to solve it (e.g. configure URL in Jenkins, set the proper headers in nginx).

The error is still showing, and I have the following lines in my jenkins.log file:

WARNING h.d.ReverseProxySetupMonitor#getTestForReverseProxySetup: https://mydomain/jenkins/manage vs. https:

I was curious why the second URL seems to be incomplete, so I looked at Chrome’s network monitor to see how the reverse proxy test was being called. It turns out the second URL is passed to the function as a parameter in the URL, in my case it was:

https://mydomain/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Fmydomain%2Fjenkins%2Fmanage/

(continued in next post as I can only include two links as new user)

The response was:

HTTP ERROR 404 https://mydomain/jenkins/manage vs. https:

URI: /jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/mydomain/jenkins/manage/

STATUS: 404

MESSAGE: https://mydomain/jenkins/manage vs. https:

SERVLET: Stapler

Powered by Jetty:// 9.4.43.v20210629

Looking at the output, URI is already missing one slash after https, and in MESSAGE the URL completely cuts off after “https:”. I believe this is a parsing error somewhere in Jenkins, and the reason why the reverse proxy test fails even though everything seems to be configured correctly.

Does this make sense, or am I missing something?

I think you’re missing something in your nginx reverse proxy configuration or in the Jenkins server name value. I’m using an nginx reverse proxy and I believe that ci.jenkins.io is also using an nginx proxy. Double check your nginx configuration.

Thanks Mark for your reply. I have doublechecked my nginx configuration and the Jenkins server name value. Everything looks fine. Here is the relevant portion of my nginx config:

   location /jenkins/ {
       proxy_pass        http://localhost:10000/jenkins/;
       proxy_redirect    default;
       proxy_set_header  Host $host:$server_port;
       proxy_set_header  X-Real-IP $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  X-Forwarded-Proto $scheme;
       proxy_set_header  X-Forwarded-Port $server_port;
       }

However, I still can’t understand why the reverse proxy test Jenkins performs is behaving the way I described. I suspect there is a bug present. Why else would the URL get mangled this way and only appear as “https:” in my Jenkins logs?

I appreciate that you believe there is a bug present. However, that then leads me to wonder why hundreds (probably thousands) of Jenkins controllers around the world are running behind nginx proxies without seeing the bug, yet you are seeing the bug. If it is a bug in Jenkins, then it is a very distinctive bug with very specific conditions that cause it to only be visible to you.

Looks like you are missing X-Forwarded-Host (which should be the external hostname) from your config. I should have spotted it when you said hostname was missing, but I just blanked on it.

Thanks for the help. I added X-Forwarded-Host to my nginx config. This didn’t help though, Jenkins is still complaining about a broken reverse proxy setup. Note that the message in the logs indicates that the inferred URL is parsed correctly from the various X-Forwarded Headers:

2022-02-02 16:03:47.264+0000 [id=6196] WARNING h.d.ReverseProxySetupMonitor#getTestForReverseProxySetup: https://mydomainname:8085/jenkins/manage vs. https:

On my nginx reverse proxy, the settings are as follows:

        location /jenkins {
                proxy_pass         http://127.0.0.1:8080;

                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_set_header   Host              $host:$server_port;
                proxy_set_header   X-Real-IP         $remote_addr;
                proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header   X-Forwarded-Proto $scheme;
                proxy_max_temp_file_size 0;

                proxy_connect_timeout      150;
                proxy_send_timeout         100;
                proxy_read_timeout         100;

                proxy_buffer_size          8k;
                proxy_buffers              4 32k;
                proxy_busy_buffers_size    64k;
                proxy_temp_file_write_size 64k;

                # Jenkins HTTP based CLI requires HTTP 1.1
                proxy_http_version       1.1;

                # JENKINS-43666 and tests confirm it helps to disable proxy_request_buffering
                proxy_request_buffering  off;

                # JENKINS-45651 notes that X-SSH-Endpoint header is not provided unless auth succeeds
                # ssh authentication for CLI will fail unless the X-SSH-Endpoint header is added
                add_header 'X-SSH-Endpoint' 'jenkins.domain.tld:50022' always;
        }

I have exactly the same problem. The error comes together with an uncaught exception.

Dec 23, 2022 9:43:47 AM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException

Caught unhandled exception with ID 3e34942d-affe-4475-8bbf-aff559e6518c
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "value" is null
	at jenkins.diagnostics.URICheckEncodingMonitor.doCheckURIEncoding(URICheckEncodingMonitor.java:46)
	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:732)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:409)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:207)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:762)
Caused: javax.servlet.ServletException
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:812)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:894)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:762)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:894)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:690)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:106)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:549)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1571)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1383)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1544)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1305)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
	at org.eclipse.jetty.server.Server.handle(Server.java:563)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:139)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:933)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1077)
	at java.base/java.lang.Thread.run(Thread.java:833)

Dec 23, 2022 9:43:48 AM WARNING hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup

https://example.com/jenkins/manage vs. https:

So the problem is the last line, it does not “register” the server name internally for some reason, just truncates the address entered in the UI to “https:”

Jenkins is receiving an empty string when it expects to receive a non-empty string value for the URI monitoring check. I suspect that your reverse proxy is incorrectly configured. The code that is generating the null pointer is also incorrect, but that doesn’t change the likely misconfiguration of your reverse proxy.

The fixEmpty converts an empty string to a null pointer, then the value.getBytes several lines later will generate a null pointer exception.

I’ve submitted the null pointer exception fix along with a test that illustrates the issue

That change does not resolve the issue in your reverse proxy configuration. It does resolve the null pointer exception that is being reported due to the issue in your reverse proxy configuration.

Thanks for the quick reply!

Currently I do not understand how my nginx proxy configuration managed to work even partly earlier.
However, this configuration works for me now (all functionality ok in jenkins), but the reverse proxy misconfiguration message is still there, jenkins thinks the server name string is empty. I will leave it at that, I can live with a red bar.

What might be a hint for where the problem is, that jenkins refuses to work with the “localhost” url, it only works if I specify the machine’s own IP. (All other services behind the same proxy work perfectly with http://localhost:port proxy_pass parameters. There is something finicky specific to jenkins.)

location ^~ /jenkins/ {
    proxy_set_header Early-Data $ssl_early_data;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-Ssl on;
    
    proxy_redirect http:// $scheme://;
    proxy_buffering off;
    proxy_request_buffering off;
    sendfile off;
    
    proxy_pass http://192.168.1.50:8080$uri$is_args$args;
}

Hi!

EDIT: This has been solved, again s##### user error. I’m leaving the rest just for anybody having the same problem and googling their way here

The root cause for this issue is indeed a configuration problem. My configuration was the following:

        # Jenkins
        location / {
            proxy_pass http://dockerserver.domain:8088/;
            proxy_redirect     default;
            proxy_http_version 1.1;

And it should be the following (remove the slash after the url):

        # Jenkins
        location / {
            proxy_pass http://dockerserver.domain:8088;
            proxy_redirect     default;
            proxy_http_version 1.1;

When the url ends with a slash, nginx will urldecode it and the Stapler no longer parses the parameters to the test correctly. More comments on the nginx behaviour here.

Thanks a lot for all the other reporters and especially to you who took your time to answer the questions.


I’m having the same issue. Jenkins reports that the reverse proxy configuration is invalid, even though I have copied the example settings for Nginx. I have a public-facing proxy server with nginx and a jenkins/jenkins:lts-jdk17 container running on a different server, exposing port for the reverse proxy.

Technically, ReverseProxySetupMonitor/test does a 302 redirect to ReverseProxySetupMonitor/testForReverseProxySetup/ and gives the referer as a parameter. Then, the latter method checks the parameter given and the referrer are the same. So far understandable.

But when I added logging for what Stapler did, i saw the following:

Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Stapler
Processing request for /administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/jenkins.domain/manage/
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(<hudson.model.Hudson@73436f> :hudson.model.Hudson,"/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(((StaplerProxy)<hudson.model.Hudson@73436f>).getTarget(),"/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(<hudson.model.Hudson@73436f>.getAdministrativeMonitor("hudson.diagnosis.ReverseProxySetupMonitor"),"/testForReverseProxySetup/https:/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(<hudson.diagnosis.ReverseProxySetupMonitor@56f94fa4> :hudson.diagnosis.ReverseProxySetupMonitor,"/testForReverseProxySetup/https:/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(((StaplerProxy)<hudson.diagnosis.ReverseProxySetupMonitor@56f94fa4>).getTarget(),"/testForReverseProxySetup/https:/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM FINE org.kohsuke.stapler.Dispatcher
-> evaluate(<hudson.diagnosis.ReverseProxySetupMonitor@56f94fa4>.getTestForReverseProxySetup("https:"),"/jenkins.domain/manage")
Sep 23, 2023 5:54:23 AM WARNING hudson.diagnosis.ReverseProxySetupMonitor getTestForReverseProxySetup
https://jenkins.domain/manage vs. https:

So, Stapler, or something before Stapler, urldecodes the parameter and starts the evaluation chain after that. The end result is, that while the parameter is correct in url (except that https:// has changed to https:/), it does not get passed as a parameter as whole - the rest of the url is left out from the call as a second parameter.

Now I’m 98% sure that this is not how it should work.

1 Like

I struggled with problem as well and reread this thread many times.

It is in an unstated requirement for the nginx examples that the proxy_pass address should be the same as the Jenkins URL set in Manage Jenkins > System.

Since the Jenkins instance often can be reached on multiple IPs, this is not a given.

Alternatively, one can use a more complex proxy_redirect.

I did google many questions on this, and when I found out the root cause (in my case), I also found out that many had made the same mistake. One practical improvement would be to add to the Jenkins nginx reverse proxy instructions something in the lines of

If nginx proxy-pass or location directive ends in slash, it will cause nginx to urldecode links and it will result in “It appears that your reverse proxy set up is broken”. If you need to end the proxy-pass or location directive with slash, check nginx documentation on how to disable url decoding.

1 Like

That seems like a good pull request for you to submit to that documentation page. Are you willing to submit the pull request?

I can try to formulate it in a concise way.