I’m running an EC2 with Docker managing jenkins/jenkins:lts which I update every few weeks. Been working fine for years, keep updating it and the plugins. Right now we’re on 2.401.3 with latest plugins.
Went to upgrade given the emergence of 2.414 and spotted the container restarting itself. OOM was the log (see below). Very odd, and I rolled back to 2.401.3-lts. The docker version is 20.10.6.
Normally the container runs with the following: JAVA_OPTS: "-Djava.awt.headless=true" - I did try with JAVA_OPTS: "-Djava.awt.headless=true -Xmx4192m" but it still crashes immediately on starting.
The EC2 itself has 8G of RAM, the container can apparently see it fine. Any ideas?
Here’s the top part from the log:
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# Can not save log file, dump to screen..
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# JVM is running with Unscaled Compressed Oops mode in which the Java heap is
# placed in the first 4GB address space. The Java Heap base address is the
# maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress
# to set the Java Heap base and to place the Java Heap above 4GB virtual address.
# This output file may be truncated or incomplete.
#
# Out of Memory Error (workerManager.hpp:70), pid=7, tid=7
#
# JRE version: (11.0.20.1+1) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.20.1+1, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to //core.7)
#
--------------- S U M M A R Y ------------
Command Line: -Duser.home=/var/jenkins_home -Djava.awt.headless=true -Djenkins.model.Jenkins.slaveAgentPort=50000 -Dhudson.lifecycle=hudson.lifecycle.ExitLifecycle /usr/share/jenkins/jenkins.war
Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 2 cores, 7G, Debian GNU/Linux 12 (bookworm)
Time: Thu Sep 21 12:22:23 2023 UTC elapsed time: 0.004346 seconds (0d 0h 0m 0s)