SpringBoot openJ9 Kubernetes Crash InterpreterEmulator.cpp:208

5/15/2020

I'm randomly getting this crash recently and I'm not sure how to debug this. Can anyone please point me in the right direction. Tools I have used are,
Spring Boot 2 + Kubernetes + adopt openj9 jdk13

Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-linux-x64-openj9/workspace/build/src/build/linux-x86_64-server-release/vm/compiler/../compiler/optimizer/InterpreterEmulator.cpp:208: 0
VMState: 0x000501ff
        unexpected bytecode in thunk archetype 0x7f98f6483ee0 at bcIndex 40 JBdload2 (41)
compiling java/lang/invoke/AsTypeHandle.invokeExact_thunkArchetype_X(Ljava/lang/Object;Ljava/lang/Object;)V at level: warm
0:/opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x798015) [0x7f9962316015]
1: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x7a2dc0) [0x7f9962320dc0]
JVMDUMP039I Processing dump event “abort”, detail “” at 2020/05/12 16:35:10 - please wait.
JVMDUMP032I JVM requested System dump using ‘/home/springboot/core.20200512.163510.1.0001.dmp’ in response to an event
-- Ronny Shibley
docker
java
kubernetes
openj9
spring-boot

2 Answers

5/28/2020

we run into the same issue, it happens by randomly using the app, no luck getting the core dump so far (Spring Boot 2.3.0, Kubernetes 1.15, adoptopenjdk/openjdk13-openj9:alpine-jre)

Error Log

Assertion failed at /home/jenkins/workspace/build-scripts/jobs/jdk13u/jdk13u-linux-x64-openj9/workspace/build/src/build/linux-x86_64-server-release/vm/compiler/../compiler/optimizer/InterpreterEmulator.cpp:208: 0 
VMState: 0x000501ff 
    unexpected bytecode in thunk archetype 0x7f0ad41a3ee0 at bcIndex 40 JBdload2 (41) 

compiling java/lang/invoke/AsTypeHandle.invokeExact_thunkArchetype_X(Ljava/lang/Object;Ljava/lang/Object;)V at level: warm 
#0: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x798015) [0x7f0aebac4015] 
#1: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x7a2dc0) [0x7f0aebacedc0] 
#2: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x465c4b) [0x7f0aeb791c4b] 
#3: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x465e3d) [0x7f0aeb791e3d] 
#4: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1f58bc) [0x7f0aeb5218bc] 
#5: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1f9532) [0x7f0aeb525532] 
#6: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1e78a0) [0x7f0aeb5138a0] 
#7: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1e9e02) [0x7f0aeb515e02] 
#8: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1cb852) [0x7f0aeb4f7852] 
#9: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1d73b9) [0x7f0aeb5033b9] 
#10: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1def1b) [0x7f0aeb50af1b] 
#11: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x55489e) [0x7f0aeb88089e] 
#12: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1d6846) [0x7f0aeb502846] 
#13: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x6164d7) [0x7f0aeb9424d7] 
#14: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x617d7b) [0x7f0aeb943d7b] 
#15: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x407f9d) [0x7f0aeb733f9d] 
#16: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x137919) [0x7f0aeb463919] 
#17: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x13861c) [0x7f0aeb46461c] 
#18: /opt/java/openjdk/lib/compressedrefs/libj9prt29.so(+0x20df3) [0x7f0af121cdf3] 
#19: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x13a115) [0x7f0aeb466115] 
#20: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x13a6bf) [0x7f0aeb4666bf] 
#21: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1366bb) [0x7f0aeb4626bb] 
#22: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x1369da) [0x7f0aeb4629da] 
#23: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x136a9a) [0x7f0aeb462a9a] 
#24: /opt/java/openjdk/lib/compressedrefs/libj9prt29.so(+0x20df3) [0x7f0af121cdf3] 
#25: /opt/java/openjdk/lib/compressedrefs/libj9jit29.so(+0x136ef4) [0x7f0aeb462ef4] 
#26: /opt/java/openjdk/lib/compressedrefs/libj9thr29.so(+0xe1a6) [0x7f0af168b1a6] 
#27: /usr/glibc-compat/lib/libpthread.so.0(+0x8f00) [0x7f0af2d9af00] 
#28: function clone+0x3f [0x7f0af2cc645f] 

JVMDUMP039I Processing dump event "abort", detail "" at 2020/05/27 06:34:58 - please wait. 
JVMDUMP032I JVM requested System dump using '//core.20200527.063458.1.0001.dmp' in response to an event 
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport %p %s %c %d %P %E" specifies that the core dump is to be piped to an external program.  Attempting to rename either core or core.71. 

JVMDUMP012E Error in System dump: The core file created by child process with pid = 71 was not found. Expected to find core file with name "//core" 
JVMDUMP032I JVM requested Java dump using '//javacore.20200527.063458.1.0002.txt' in response to an event 
JVMDUMP010I Java dump written to //javacore.20200527.063458.1.0002.txt 
JVMDUMP032I JVM requested Snap dump using '//Snap.20200527.063458.1.0003.trc' in response to an event 
JVMDUMP010I Snap dump written to //Snap.20200527.063458.1.0003.trc 
JVMDUMP007I JVM Requesting JIT dump using '//jitdump.20200527.063458.1.0004.dmp' 
JVMDUMP010I JIT dump written to //jitdump.20200527.063458.1.0004.dmp 
JVMDUMP013I Processed dump event "abort", detail "". 

Submitted issue on openj9 github

-- kimminger
Source: StackOverflow

5/28/2020

TLDR: This is fixed in the v.0.20.0 release (April 2020) of OpenJ9. The latest release binaries from AdoptOpenJDK for JDK 8, 11 & 14 contain the fix.

The assertion failure is an assertion JIT compiling the MethodHandle (MH) code. A new abstract interpreter was added to give the JIT a better view of the MH "thunkArchetype" code and it missed some of the bytecodes that could occur in MH thunkArchetypes.

See https://github.com/eclipse/openj9/pull/8598 for more details of the fix.

-- Dan Heidinga
Source: StackOverflow