HTTP 503 Errors After Upgrading from 2.319.3 to 2.332.1

Using Jenkins
1

My Jenkins users are reporting frequent occurrences of 503 errors (Service unavailable) when using the Jenkins UI. The Jenkins log does not show any WARNINGS or SEVERE entries when the issue occurs. The issue doesn’t seem to effect the builds and only the UI.

I have the Jenkins.war loaded in a Windows server 2019 1809 based docker container running on a docker SWARM of 4 hosts, each using the latest version of docker for Windows Server 1809 (docker EE version 20.10.9). I also have a Traefik controller container configured on the front end of the SWARM, that routes HTTP/HTTPS traffic from a company LB to the Jenkins controller container backend.

The Jenkins container is using the openjdk11 JVM, with no explicit JVM options added (default JVM config for memory management of the heap, G1GC). The docker host has 64GB of memory so the JVM default sizing of the JVM heap is 1/4 of 64GB, or 16GB. My current thoughts are that maybe there are intermittent ‘stop the world’ garbage collections going on, but feel that the builds would also be having problems if that was the root cause. Also am using the AD plugin for user authentication connected to 3 domains DCs. I recently changed the authentication from LDAP to AD since the LDAP plugin was seeing way too many errors in the Jenkins log.

Anyway, I thought I would post this issue on this forum to see if anyone else who recently upgraded to 2.332.1 might also be seeing the same issue.

2

May be worth comparing your JVM settings to the settings recommended by the support team at CloudBees:

In the garbage collection specifications, it says:

Currently it is recommended to use the G1 Garbage Collection algorithm.

G1 is superior to ParallelGC or CMS for a Jenkins JVM. This is because G1 avoids long “stop the world” pauses which can take many seconds in a large collection. Jenkins is an application that relies on low latency, so the shorter the GC pauses, the better. It is also the default Garbage Collector in Java 9.

Specify G1 GC as follows:

-XX:+UseG1GC

The G1 garbage collector is the default on Java 11. I would expect that it is the garbage collector you’re already using.