Nginx + Jenkins: it appears that your reverse proxy set up is broken

Using Jenkins Ask a question
1

Jenkins setup:

  1. Script output:

Jenkins: 2.426.1
OS: Linux - 5.15.0-79-generic
Java: 17.0.9 - Private Build (OpenJDK 64-Bit Server VM)

ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
bootstrap5-api:5.3.2-2
bouncycastle-api:2.29
branch-api:2.1135.v8de8e7899051
build-timeout:1.31
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.2
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-94.v3e1f4a_926e49
credentials:1309.v8835d63eb_d8a_
credentials-binding:642.v737c34dea_6c2
display-url-api:2.200.vb_9327d658781
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.3-1
email-ext:2.102
font-awesome-api:6.4.2-1
git:5.2.1
git-client:4.6.0
github:1.37.3.1
github-api:1.318-461.v7a_c09c9fa_d63
github-branch-source:1752.vc201a_0235d80
gitlab-plugin:1.7.16
gradle:2.9
instance-identity:185.v303dc7c645f9
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.3-372.v309620682326
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jersey2-api:2.41-133.va_03323b_a_1396
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.7.1-1
jsch:0.2.8-65.v052c39de79b_2
junit:1240.vf9529b_881428
ldap:711.vb_d1a_491714dc
mailer:463.vedf8358e006b_
matrix-auth:3.2.1
matrix-project:818.v7eb_e657db_924
mina-sshd-api-common:2.11.0-86.v836f585d47fa_
mina-sshd-api-core:2.11.0-86.v836f585d47fa_
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
pipeline-build-step:516.v8ee60a_81c5b_9
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:689.veec561a_dee13
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2151.ve32c9d209a_3f
pipeline-model-definition:2.2151.ve32c9d209a_3f
pipeline-model-extensions:2.2151.ve32c9d209a_3f
pipeline-rest-api:2.34
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2151.ve32c9d209a_3f
pipeline-stage-view:2.34
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.6.0
publish-over:0.22
publish-over-ssh:1.25
resource-disposer:0.23
scm-api:683.vb_16722fb_b_80b_
script-security:1294.v99333c047434
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
ssh-steps:2.0.68.va_d21a_12a_6476
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
timestamper:1.26
token-macro:384.vf35b_f26814ec
trilead-api:2.84.v72119de229b_7
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1283.v99c10937efcb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3812.vc3031a_b_a_c955
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1360.vc6700e3136f5
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:865.v43e78cc44e0d
ws-cleanup:0.45

  1. Running Jenkins directly
  2. Accessed through reverse proxy
  3. Installed via deb
  4. Involved agents are run via SSH
  5. Web browser - Chrome latest.

Jenkins server works behind the reverse proxy server, so:

user`s request → reverse proxy server nginx → jenkins server nginx → Jenkins itself

Getting this message on the Jenkins configuration page: “It appears that your reverse proxy set up is broken” .

Reverse proxy nginx configuration:

server {                                                                                              
    server_name jenkins.domain.com;

    location / {
        proxy_pass       http://jenkins_server_ip:80;                                                             
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/jenkins.domain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/jenkins.domain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Jenkins server nginx configuration:

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_pass            http://127.0.0.1:8080;
        proxy_set_header      Host $host;
        proxy_set_header      X-Real-IP $remote_addr;
        proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto $scheme;
        proxy_set_header      X-Forwarded-Host $host;
        proxy_set_header      X-Forwarded-Port $server_port;
        proxy_set_header      Upgrade $http_upgrade;
        proxy_set_header      Connection $connection_upgrade;

        proxy_http_version 1.1; 
        proxy_request_buffering off;
    }
}

After trying administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test tool with curl a warning appears in Jenkins` log:

http://jenkins.domain.com/manage vs. https:

It seems like for some reason it is not able to form a full URL

Additional question: is it possible to make Jenkins report why exactly it considers reverse proxy set up is broken?

2

why 2 reverse proxy servers?

3

Don`t know for sure, it was not made by me. As I understand, people who made it thought that it will increase security.

4

The opposite is true I would say. The traffic between the 2 nginx instances is unencrypted. Not sure if they run on the same host, then the traffic might not leave the host.
But it allows to access jenkins via port 80 without ssl

5

To the ones who may face the same problem. Here is how I resolved this issue:

It turned out that Nginx variables like $scheme, $server_port and so on are initialized using the URL of the request and not the forwarded headers. That is why Nginx on the Jenkins server was getting “http” and “80” instead of “https” and “443”.
In order to fix it I just hardcoded these values in the configuration:

    listen 80;
    server_name jenkins.domain.com;

    location / {
        proxy_pass            http://127.0.0.1:8080;

        proxy_set_header      Host $host;
        proxy_set_header      X-Real-IP $remote_addr;
        proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header      X-Forwarded-Proto https;
        proxy_set_header      X-Forwarded-Host $host;
        proxy_set_header      X-Forwarded-Port 443;
        proxy_set_header      Upgrade $http_upgrade;
        proxy_set_header      Connection $connection_upgrade;
    }
}```