Bug 1316574 - Random JVM crash: /usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0
Random JVM crash: /usr/lib/jvm/java-1.8.0/bin/java': double free or corruptio...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: java-1.8.0-openjdk (Show other bugs)
7.0
x86_64 Linux
unspecified Severity medium
: rc
: ---
Assigned To: Deepak Bhole
BaseOS QE - Apps
https://bugs.openjdk.java.net/browse/...
:
Depends On:
Blocks: 1298243
  Show dependency treegraph
 
Reported: 2016-03-10 09:05 EST by Andrey Loskutov
Modified: 2016-12-02 12:33 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-02 12:33:00 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
JVM crash dump (45.20 KB, text/plain)
2016-03-10 09:05 EST, Andrey Loskutov
no flags Details
Output written to the console on crash (74.62 KB, patch)
2016-03-10 09:05 EST, Andrey Loskutov
no flags Details | Diff

  None (edit)
Description Andrey Loskutov 2016-03-10 09:05:08 EST
Created attachment 1134897 [details]
JVM crash dump

Description of problem:
The headless Eclipse application using java 8 open jdk crashed during compilation of xtext/java related code.

The eclipse 3.8.2 was started with the eclipse native launcher, command line for the eclipse:

-vm /usr/lib/jvm/java-1.8.0/bin/java  -nosplash --launcher.suppressErrors -application com.advantest.itee.stpl.standalone.stplstandalone -configuration /TestBed/tplc/_jenkinsbs/_configuration -vmargs -Xmx4096m -Xss5m -Xverify:none -XX:CompileThreshold=1000 -XX:ReservedCodeCacheSize=256m -XX:ErrorFile=/TestBed/workspace/tplc_vm_crash_%p.log -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -Dosgi.bundlefile.limit=100 -Dxtext.qn.interning=true

Version-Release number of selected component (if applicable):
$ rpm -qa | grep java-1.8.0
java-1.8.0-openjdk-headless-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-devel-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-src-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-src-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-debuginfo-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-headless-1.8.0.71-2.b15.el7_2.x86_64

Steps to Reproduce:
We do not have steps to reproduce yet.
The code basically starts Eclipse without workbench (UI) and runs Java and Xtext based compiler over few files.

Actual results:

The execution terminated after few seconds with the error below:

*** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 ***
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe5aa133e75, pid=11123, tid=140622636713728
#
# JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V======= Backtrace: =========
  [libjvm.so+0x5bae75]/lib64/libc.so.6(+0x7d023)[0x7fe5aaaf1023]

Expected results:
No crash

Additional info:
I'm attaching the *incompleted* crash file written by the JVM and the output printed to the console at same time.
Comment 1 Andrey Loskutov 2016-03-10 09:05 EST
Created attachment 1134898 [details]
Output written to the console on crash
Comment 2 Andrey Loskutov 2016-03-10 09:10:57 EST
I know that this is nearly impossible to fix anything without steps to reproduce and only few bits of the crash log. Nevertheless, any hints what could be a root cause / possible workarounds are welcome. Since this happened during our integration tests, I hope to get some more data once this occurs again (we run thousands of tests which do this compile tasks every night).
Comment 4 Andrey Loskutov 2016-04-21 08:43:08 EDT
Meanwhile we saw 18 crashes in last 4 weeks, and I've analyzed 12 test log files I could collect.

11 crashes do not leave an error report file. The one report file is attached here is incomplete.

All crashes have in common: they report same memory address (0x00000007c0036fc0) and have exact same backtrace pointing to the libjvm.so (except the actual memory addresses in square brackets):

*** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d023)[0x7f406fd7c023]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x5e1821)[0x7f406f3e5821]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x904d2f)[0x7f406f708d2f]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x9056e5)[0x7f406f7096e5]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0d5ba)[0x7f406f8115ba]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0da21)[0x7f406f811a21]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x869702)[0x7f406f66d702]
/lib64/libpthread.so.0(+0x7dc5)[0x7f40706f0dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f406fdf528d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00600000-00601000 r--p 00000000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00601000-00602000 rw-p 00001000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00a3e000-00a5f000 rw-p 00000000 00:00 0                                  [heap]
6c0000000-6d4380000 rw-p 00000000 00:00 0 
6d4380000-76ab00000 ---p 00000000 00:00 0 
76ab00000-7bfc80000 rw-p 00000000 00:00 0 
7bfc80000-7c0000000 ---p 00000000 00:00 0 
7c0000000-7c08aa000 rw-p 00000000 00:00 0 
7c08aa000-800000000 ---p 00000000 00:00 0 

The tests where this application is involved run ~100 times per day, and each calls the application ~150 times, resulting in ~15000 invocations per day.
For the time period of 6 weeks we executed this application ~15000*6*7 times (let say 600000 times) and observed 18 crashes.
This results in 0.003% of failures.
I also run the same application in a loop over 5000 times with not a single crash.
Meanwhile I've reduced VM arguments to only -Xmx4096m -Xss5m -XX:ErrorFile=/path_to_test_dir/tplc_vm_crash_%p.log.

I would appreciate if anyone with a debug symbols could check the backtrace above and also this strange memory address (0x00000007c0036fc0) and may be give a hint how to proceed from here.
Comment 5 Andrey Loskutov 2016-04-21 08:55:37 EDT
I've created RH case #01621685 referencing to this issue.
Comment 6 Andrey Loskutov 2016-07-21 02:31:19 EDT
This bug can be closed. After update to OpenJdk 1.8.0_91 the crashes disappeared, most likely due the fix of  https://bugs.openjdk.java.net/browse/JDK-8081283 (we were using 1.8.0.71 previously).
Comment 8 Deepak Bhole 2016-12-02 12:33:00 EST
Closing per comment #6

Note You need to log in before you can comment on or make changes to this bug.