RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1316574 - Random JVM crash: /usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0
Summary: Random JVM crash: /usr/lib/jvm/java-1.8.0/bin/java': double free or corruptio...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: java-1.8.0-openjdk
Version: 7.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Deepak Bhole
QA Contact: BaseOS QE - Apps
URL: https://bugs.openjdk.java.net/browse/...
Whiteboard:
Depends On:
Blocks: 1298243
TreeView+ depends on / blocked
 
Reported: 2016-03-10 14:05 UTC by Andrey Loskutov
Modified: 2019-11-14 07:34 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-12-02 17:33:00 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
JVM crash dump (45.20 KB, text/plain)
2016-03-10 14:05 UTC, Andrey Loskutov 		no flags Details
Output written to the console on crash (74.62 KB, patch)
2016-03-10 14:05 UTC, Andrey Loskutov 		no flags Details | Diff
View All

Description Andrey Loskutov 2016-03-10 14:05:08 UTC 
Created attachment 1134897 [details]
JVM crash dump

Description of problem:
The headless Eclipse application using java 8 open jdk crashed during compilation of xtext/java related code.

The eclipse 3.8.2 was started with the eclipse native launcher, command line for the eclipse:

-vm /usr/lib/jvm/java-1.8.0/bin/java  -nosplash --launcher.suppressErrors -application com.advantest.itee.stpl.standalone.stplstandalone -configuration /TestBed/tplc/_jenkinsbs/_configuration -vmargs -Xmx4096m -Xss5m -Xverify:none -XX:CompileThreshold=1000 -XX:ReservedCodeCacheSize=256m -XX:ErrorFile=/TestBed/workspace/tplc_vm_crash_%p.log -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -Dosgi.bundlefile.limit=100 -Dxtext.qn.interning=true

Version-Release number of selected component (if applicable):
$ rpm -qa | grep java-1.8.0
java-1.8.0-openjdk-headless-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-devel-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-src-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-src-debug-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-debuginfo-1.8.0.71-2.b15.el7_2.x86_64
java-1.8.0-openjdk-headless-1.8.0.71-2.b15.el7_2.x86_64

Steps to Reproduce:
We do not have steps to reproduce yet.
The code basically starts Eclipse without workbench (UI) and runs Java and Xtext based compiler over few files.

Actual results:

The execution terminated after few seconds with the error below:

*** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 ***
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fe5aa133e75, pid=11123, tid=140622636713728
#
# JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15)
# Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V======= Backtrace: =========
  [libjvm.so+0x5bae75]/lib64/libc.so.6(+0x7d023)[0x7fe5aaaf1023]

Expected results:
No crash

Additional info:
I'm attaching the *incompleted* crash file written by the JVM and the output printed to the console at same time.
Comment 1 Andrey Loskutov 2016-03-10 14:05:45 UTC 
Created attachment 1134898 [details]
Output written to the console on crash
Comment 2 Andrey Loskutov 2016-03-10 14:10:57 UTC 
I know that this is nearly impossible to fix anything without steps to reproduce and only few bits of the crash log. Nevertheless, any hints what could be a root cause / possible workarounds are welcome. Since this happened during our integration tests, I hope to get some more data once this occurs again (we run thousands of tests which do this compile tasks every night).
Comment 4 Andrey Loskutov 2016-04-21 12:43:08 UTC 
Meanwhile we saw 18 crashes in last 4 weeks, and I've analyzed 12 test log files I could collect.

11 crashes do not leave an error report file. The one report file is attached here is incomplete.

All crashes have in common: they report same memory address (0x00000007c0036fc0) and have exact same backtrace pointing to the libjvm.so (except the actual memory addresses in square brackets):

*** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7d023)[0x7f406fd7c023]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x5e1821)[0x7f406f3e5821]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x904d2f)[0x7f406f708d2f]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x9056e5)[0x7f406f7096e5]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0d5ba)[0x7f406f8115ba]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0da21)[0x7f406f811a21]
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x869702)[0x7f406f66d702]
/lib64/libpthread.so.0(+0x7dc5)[0x7f40706f0dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f406fdf528d]
======= Memory map: ========
00400000-00401000 r-xp 00000000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00600000-00601000 r--p 00000000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00601000-00602000 rw-p 00001000 fd:01 6575573264                         /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java
00a3e000-00a5f000 rw-p 00000000 00:00 0                                  [heap]
6c0000000-6d4380000 rw-p 00000000 00:00 0 
6d4380000-76ab00000 ---p 00000000 00:00 0 
76ab00000-7bfc80000 rw-p 00000000 00:00 0 
7bfc80000-7c0000000 ---p 00000000 00:00 0 
7c0000000-7c08aa000 rw-p 00000000 00:00 0 
7c08aa000-800000000 ---p 00000000 00:00 0 

The tests where this application is involved run ~100 times per day, and each calls the application ~150 times, resulting in ~15000 invocations per day.
For the time period of 6 weeks we executed this application ~15000*6*7 times (let say 600000 times) and observed 18 crashes.
This results in 0.003% of failures.
I also run the same application in a loop over 5000 times with not a single crash.
Meanwhile I've reduced VM arguments to only -Xmx4096m -Xss5m -XX:ErrorFile=/path_to_test_dir/tplc_vm_crash_%p.log.

I would appreciate if anyone with a debug symbols could check the backtrace above and also this strange memory address (0x00000007c0036fc0) and may be give a hint how to proceed from here.
Comment 5 Andrey Loskutov 2016-04-21 12:55:37 UTC 
I've created RH case #01621685 referencing to this issue.
Comment 6 Andrey Loskutov 2016-07-21 06:31:19 UTC 
This bug can be closed. After update to OpenJdk 1.8.0_91 the crashes disappeared, most likely due the fix of  https://bugs.openjdk.java.net/browse/JDK-8081283 (we were using 1.8.0.71 previously).
Comment 8 Deepak Bhole 2016-12-02 17:33:00 UTC 
Closing per comment #6
Note You need to log in before you can comment on or make changes to this bug.