Reverse Proxy Test does not handle URL correctly

johannes · January 20, 2022, 4:12pm

Hello,

I am using Jenkins 2.319.1 with nginx as reverse proxy. There is a message in Jenkins that “it appears that your reverse proxy set up is broken”. I researched this problem quite a bit, and followed the usual steps to solve it (e.g. configure URL in Jenkins, set the proper headers in nginx).

The error is still showing, and I have the following lines in my jenkins.log file:

WARNING h.d.ReverseProxySetupMonitor#getTestForReverseProxySetup: https://mydomain/jenkins/manage vs. https:

I was curious why the second URL seems to be incomplete, so I looked at Chrome’s network monitor to see how the reverse proxy test was being called. It turns out the second URL is passed to the function as a parameter in the URL, in my case it was:

https://mydomain/jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https%3A%2F%2Fmydomain%2Fjenkins%2Fmanage/

(continued in next post as I can only include two links as new user)

johannes · January 20, 2022, 8:05pm

The response was:

HTTP ERROR 404 https://mydomain/jenkins/manage vs. https:

URI: /jenkins/administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/testForReverseProxySetup/https:/mydomain/jenkins/manage/

STATUS: 404

MESSAGE: https://mydomain/jenkins/manage vs. https:

SERVLET: Stapler

Powered by Jetty:// 9.4.43.v20210629

Looking at the output, URI is already missing one slash after https, and in MESSAGE the URL completely cuts off after “https:”. I believe this is a parsing error somewhere in Jenkins, and the reason why the reverse proxy test fails even though everything seems to be configured correctly.

Does this make sense, or am I missing something?

MarkEWaite · January 21, 2022, 1:44am

I think you’re missing something in your nginx reverse proxy configuration or in the Jenkins server name value. I’m using an nginx reverse proxy and I believe that ci.jenkins.io is also using an nginx proxy. Double check your nginx configuration.

johannes · February 1, 2022, 9:09am

Thanks Mark for your reply. I have doublechecked my nginx configuration and the Jenkins server name value. Everything looks fine. Here is the relevant portion of my nginx config:

   location /jenkins/ {
       proxy_pass        http://localhost:10000/jenkins/;
       proxy_redirect    default;
       proxy_set_header  Host $host:$server_port;
       proxy_set_header  X-Real-IP $remote_addr;
       proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
       proxy_set_header  X-Forwarded-Proto $scheme;
       proxy_set_header  X-Forwarded-Port $server_port;
       }

However, I still can’t understand why the reverse proxy test Jenkins performs is behaving the way I described. I suspect there is a bug present. Why else would the URL get mangled this way and only appear as “https:” in my Jenkins logs?

MarkEWaite · February 1, 2022, 7:14pm

I appreciate that you believe there is a bug present. However, that then leads me to wonder why hundreds (probably thousands) of Jenkins controllers around the world are running behind nginx proxies without seeing the bug, yet you are seeing the bug. If it is a bug in Jenkins, then it is a very distinctive bug with very specific conditions that cause it to only be visible to you.

halkeye · February 1, 2022, 9:18pm

github.com

jenkinsci/jenkins/blob/59ab7d299c87ab26afc2ab315686998f20f7b983/core/src/main/java/hudson/diagnosis/ReverseProxySetupMonitor.java

/*
 * The MIT License
 *
 * Copyright (c) 2010, InfraDNA, Inc.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,

This file has been truncated. show original

github.com

jenkinsci/jenkins/blob/59ab7d299c87ab26afc2ab315686998f20f7b983/core/src/main/java/jenkins/model/Jenkins.java#L2492

      
        
             * correctly, especially when a migration is involved), but the downside
             * is that unless you are processing a request, this method doesn't work.
             *
             * <p>Please note that this will not work in all cases if Jenkins is running behind a
             * reverse proxy which has not been fully configured.
             * Specifically the {@code Host} and {@code X-Forwarded-Proto} headers must be set.
             * <a href="https://www.jenkins.io/doc/book/system-administration/reverse-proxy-configuration-apache/">Reverse proxy - Apache</a>
             * shows some examples of configuration.
             * @since 1.263
             */
            public @NonNull String getRootUrlFromRequest() {
                StaplerRequest req = Stapler.getCurrentRequest();
                if (req == null) {
                    throw new IllegalStateException("cannot call getRootUrlFromRequest from outside a request handling thread");
                }
                StringBuilder buf = new StringBuilder();
                String scheme = getXForwardedHeader(req, "X-Forwarded-Proto", req.getScheme());
                buf.append(scheme).append("://");
                String host = getXForwardedHeader(req, "X-Forwarded-Host", req.getServerName());
                int index = host.lastIndexOf(':');
                int port = req.getServerPort();

Looks like you are missing X-Forwarded-Host (which should be the external hostname) from your config. I should have spotted it when you said hostname was missing, but I just blanked on it.

johannes · February 2, 2022, 4:18pm

Thanks for the help. I added X-Forwarded-Host to my nginx config. This didn’t help though, Jenkins is still complaining about a broken reverse proxy setup. Note that the message in the logs indicates that the inferred URL is parsed correctly from the various X-Forwarded Headers:

2022-02-02 16:03:47.264+0000 [id=6196] WARNING h.d.ReverseProxySetupMonitor#getTestForReverseProxySetup: https://mydomainname:8085/jenkins/manage vs. https:

github.com

jenkinsci/jenkins/blob/59ab7d299c87ab26afc2ab315686998f20f7b983/core/src/main/java/hudson/diagnosis/ReverseProxySetupMonitor.java#L102

      
        
            @Restricted(DoNotUse.class) // WebOnly
            @RestrictedSince("2.235")
            @StaplerDispatchable
            public void getTestForReverseProxySetup(String rest) {
                Jenkins j = Jenkins.get();
                String inferred = j.getRootUrlFromRequest() + "manage";
                // TODO this could also verify that j.getRootUrl() has been properly configured, and send a different message if not
                if (rest.startsWith(inferred)) { // not using equals due to JENKINS-24014
                    throw HttpResponses.ok();
                } else {
                    LOGGER.log(Level.WARNING, "{0} vs. {1}", new Object[] {inferred, rest});
                    throw HttpResponses.errorWithoutStack(404, inferred + " vs. " + rest);
                }
            }
            
            
@Override
            public Permission getRequiredPermission() {
                return Jenkins.SYSTEM_READ;
            }
            
            
/**

MarkEWaite · February 2, 2022, 8:37pm

On my nginx reverse proxy, the settings are as follows:

        location /jenkins {
                proxy_pass         http://127.0.0.1:8080;

                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_set_header   Host              $host:$server_port;
                proxy_set_header   X-Real-IP         $remote_addr;
                proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
                proxy_set_header   X-Forwarded-Proto $scheme;
                proxy_max_temp_file_size 0;

                proxy_connect_timeout      150;
                proxy_send_timeout         100;
                proxy_read_timeout         100;

                proxy_buffer_size          8k;
                proxy_buffers              4 32k;
                proxy_busy_buffers_size    64k;
                proxy_temp_file_write_size 64k;

                # Jenkins HTTP based CLI requires HTTP 1.1
                proxy_http_version       1.1;

                # JENKINS-43666 and tests confirm it helps to disable proxy_request_buffering
                proxy_request_buffering  off;

                # JENKINS-45651 notes that X-SSH-Endpoint header is not provided unless auth succeeds
                # ssh authentication for CLI will fail unless the X-SSH-Endpoint header is added
                add_header 'X-SSH-Endpoint' 'jenkins.domain.tld:50022' always;
        }