The metrics plugin reports a segmentation error causing Jenkins coredump

Using Jenkins Ask a question
,
1

My Jenkins experienced a core dump without any configuration changes. Upon inspecting the hs_err_xxx.log file, I identified a thread encountering a memory segmentation fault, resulting in a null pointer exception.

The log indicated: "JavaThread “Metrics-HealthChecks-155622”.

java version:
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment Temurin-11.0.16.1+1 (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)

Jenkins version: Jenkins 2.414.1

Metrics plugin version: 4.2.18-442.v02e107157925

The memory location where the problem occurred is: 0x00007f7362c9b598
J 5182957 c2 java.lang.invoke.VarHandleObjects$FieldInstanceReadWrite.compareAndSet(Ljava/lang/invoke/VarHandleObjects$FieldInstanceReadWrite;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Z java.base@11.0.16.1 (38 bytes) @ 0x00007f7362c9b598 [0x00007f7362c9b540+0x0000000000000058]

A hidden null pointer appears at this memory address:
Event: 2334409.135 Thread 0x00007f48e80ac800 Implicit null exception at 0x00007f7362c9b598 to 0x0000000000000000

Check the current thread position 0x00007f48e80ac800:
Current thread (0x00007f48e80ac800): JavaThread “Metrics-HealthChecks-155622” daemon [_thread_in_Java, id=108274, stack(0x00007f4dba9f0000,0x00007f4dbaaf1000)]

A memory segmentation fault occurs:
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000003d

2

I checked the release note of subsequent versions of the metrics plugin and found no fix for this problem. Is this a known issue?

3

You’re running with a 2 year old java version. And also the Jenkins version is a bit older.
You should check first if the problem is reproducible in current Jenkins and an up-to-date Java version

1 Like
4

Hi @mawinter69 ，does Jenkins support Java 22 now? I am considering upgrading the version recently, but I only see that Jenkins changelog mentions the addition of Java 11 support.

I am considering upgrading Jenkins to version 2.440.3

5

See Java Support Policy and Introducing the 2 + 2 + 2 Java support plan for the supported Java versions.

1 Like
6

@mawinter69 I wanted to give you an update on the issue I reported a while back. As suggested, I upgraded to Jenkins version 2.440.3 and Java version 21.0.2, and for a while, everything was smooth sailing. However, half a year later, I’m experiencing the same problem again. Here’s the error log for your reference:

# JRE version: OpenJDK Runtime Environment (21.0.2+13) (build 21.0.2+13-58)
# Java VM: OpenJDK 64-Bit Server VM (21.0.2+13-58, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xcb4f88]  OopStorage::Block::release_entries(unsigned long, OopStorage*)+0x38
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping t
o /var/lib/jenkins/core.86417)
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xmx256g -XX:+AlwaysPreTouch -XX:+UseStringDeduplication -XX:+ParallelRefProcEnabled -XX:+DisableExplicitGC -Djava.awt.headless=true -Dmail.smtp.st
arttls.enable=true -Dmail.smtp.ssl.protocols=TLSv1.2 /usr/share/java/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8888

Host: Intel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz, 128 cores, 503G, Ubuntu 18.04.6 LTS
Time: Wed Dec 18 14:59:51 2024 CST elapsed time: 3608396.639533 seconds (41d 18h 19m 56s)

---------------  T H R E A D  ---------------

Current thread (0x00007f9ff032fda0):  JavaThread "Monitor Deflation Thread" daemon [_thread_in_vm, id=86567, stack(0x00007f5dbcf41000,0x00007f5dbd042000) (1028K)
]

Stack: [0x00007f5dbcf41000,0x00007f5dbd042000],  sp=0x00007f5dbd040930,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xcb4f88]  OopStorage::Block::release_entries(unsigned long, OopStorage*)+0x38
V  [libjvm.so+0xcb51f9]  OopStorage::release(oopDesc* const*)+0x89
V  [libjvm.so+0xc9f3d8]  ObjectMonitor::deflate_monitor()+0x248
V  [libjvm.so+0xe4d2b9]  ObjectSynchronizer::deflate_monitor_list(Thread*, LogStream*, elapsedTimer*, ObjectMonitorsHashtable*)+0x129
V  [libjvm.so+0xe4d474]  ObjectSynchronizer::deflate_idle_monitors(ObjectMonitorsHashtable*)+0x174
V  [libjvm.so+0xc6fee7]  MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*)+0x137
V  [libjvm.so+0x8f8a78]  JavaThread::thread_main_inner() [clone .part.0]+0xb8
V  [libjvm.so+0xe9e998]  Thread::call_run()+0xa8
V  [libjvm.so+0xcc444a]  thread_native_entry(Thread*)+0xda

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000238

Registers:
RAX=0x0000000000000001, RBX=0x00007f9ff002b7a0, RCX=0x00000fe9ed87590e, RDX=0x00007f9ff002b760
RSP=0x00007f5dbd040930, RBP=0x00007f5dbd040a10, RSI=0x0000000000004000, RDI=0x0000000000000000
R8 =0x0000000000000000, R9 =0x00007f5dbd040bc0, R10=0x0000000000000000, R11=0x0000000000000000
R12=0x0000000000004000, R13=0x0000000000000000, R14=0x00007f9ff002b760, R15=0x0000000000000238
RIP=0x00007f9ff69a0f88, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e


Register to memory mapping:

RAX=0x0000000000000001 is an unknown value
RBX=0x00007f9ff002b7a0 points into unknown readable memory: 0x00000000000001de | de 01 00 00 00 00 00 00
RCX=0x00000fe9ed87590e is an unknown value
RDX=0x00007f9ff002b760 points into unknown readable memory: 0x00007f9ff0002770 | 70 27 00 f0 9f 7f 00 00
RSP=0x00007f5dbd040930 is pointing into the stack for thread: 0x00007f9ff032fda0
RBP=0x00007f5dbd040a10 is pointing into the stack for thread: 0x00007f9ff032fda0
RSI=0x0000000000004000 is an unknown value
RDI=0x0 is null
R8 =0x0 is null
R9 =0x00007f5dbd040bc0 is pointing into the stack for thread: 0x00007f9ff032fda0
R10=0x0 is null
R11=0x0 is null
R12=0x0000000000004000 is an unknown value
R13=0x0 is null
R14=0x00007f9ff002b760 points into unknown readable memory: 0x00007f9ff0002770 | 70 27 00 f0 9f 7f 00 00
R15=0x0000000000000238 is an unknown value

I’ve tried to follow the steps you guys provided earlier, but it seems like we might need to dive deeper into this. Could you offer some more insights or suggest any additional steps I could take to troubleshoot this further? I’d really appreciate any help the community can provide.

7

Again the latest java 21 version is 21.0.5 and the latest Jenkins version is 2.479.2
You should regularly update Jenkins and Java