| Summary: | Random JVM crash: /usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Andrey Loskutov <loskutov> | ||||||
| Component: | java-1.8.0-openjdk | Assignee: | Deepak Bhole <dbhole> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | BaseOS QE - Apps <qe-baseos-apps> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 7.0 | CC: | cww, dkochuka, jvanek | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| URL: | https://bugs.openjdk.java.net/browse/JDK-8081283 | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2016-12-02 17:33:00 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1298243 | ||||||||
| Attachments: |
|
||||||||
Created attachment 1134898 [details]
Output written to the console on crash
I know that this is nearly impossible to fix anything without steps to reproduce and only few bits of the crash log. Nevertheless, any hints what could be a root cause / possible workarounds are welcome. Since this happened during our integration tests, I hope to get some more data once this occurs again (we run thousands of tests which do this compile tasks every night). Meanwhile we saw 18 crashes in last 4 weeks, and I've analyzed 12 test log files I could collect. 11 crashes do not leave an error report file. The one report file is attached here is incomplete. All crashes have in common: they report same memory address (0x00000007c0036fc0) and have exact same backtrace pointing to the libjvm.so (except the actual memory addresses in square brackets): *** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7d023)[0x7f406fd7c023] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x5e1821)[0x7f406f3e5821] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x904d2f)[0x7f406f708d2f] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x9056e5)[0x7f406f7096e5] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0d5ba)[0x7f406f8115ba] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0xa0da21)[0x7f406f811a21] /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/jre/lib/amd64/server/libjvm.so(+0x869702)[0x7f406f66d702] /lib64/libpthread.so.0(+0x7dc5)[0x7f40706f0dc5] /lib64/libc.so.6(clone+0x6d)[0x7f406fdf528d] ======= Memory map: ======== 00400000-00401000 r-xp 00000000 fd:01 6575573264 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java 00600000-00601000 r--p 00000000 fd:01 6575573264 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java 00601000-00602000 rw-p 00001000 fd:01 6575573264 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64/bin/java 00a3e000-00a5f000 rw-p 00000000 00:00 0 [heap] 6c0000000-6d4380000 rw-p 00000000 00:00 0 6d4380000-76ab00000 ---p 00000000 00:00 0 76ab00000-7bfc80000 rw-p 00000000 00:00 0 7bfc80000-7c0000000 ---p 00000000 00:00 0 7c0000000-7c08aa000 rw-p 00000000 00:00 0 7c08aa000-800000000 ---p 00000000 00:00 0 The tests where this application is involved run ~100 times per day, and each calls the application ~150 times, resulting in ~15000 invocations per day. For the time period of 6 weeks we executed this application ~15000*6*7 times (let say 600000 times) and observed 18 crashes. This results in 0.003% of failures. I also run the same application in a loop over 5000 times with not a single crash. Meanwhile I've reduced VM arguments to only -Xmx4096m -Xss5m -XX:ErrorFile=/path_to_test_dir/tplc_vm_crash_%p.log. I would appreciate if anyone with a debug symbols could check the backtrace above and also this strange memory address (0x00000007c0036fc0) and may be give a hint how to proceed from here. I've created RH case #01621685 referencing to this issue. This bug can be closed. After update to OpenJdk 1.8.0_91 the crashes disappeared, most likely due the fix of https://bugs.openjdk.java.net/browse/JDK-8081283 (we were using 1.8.0.71 previously). Closing per comment #6 |
Created attachment 1134897 [details] JVM crash dump Description of problem: The headless Eclipse application using java 8 open jdk crashed during compilation of xtext/java related code. The eclipse 3.8.2 was started with the eclipse native launcher, command line for the eclipse: -vm /usr/lib/jvm/java-1.8.0/bin/java -nosplash --launcher.suppressErrors -application com.advantest.itee.stpl.standalone.stplstandalone -configuration /TestBed/tplc/_jenkinsbs/_configuration -vmargs -Xmx4096m -Xss5m -Xverify:none -XX:CompileThreshold=1000 -XX:ReservedCodeCacheSize=256m -XX:ErrorFile=/TestBed/workspace/tplc_vm_crash_%p.log -Djava.awt.headless=true -Djava.util.Arrays.useLegacyMergeSort=true -Dosgi.bundlefile.limit=100 -Dxtext.qn.interning=true Version-Release number of selected component (if applicable): $ rpm -qa | grep java-1.8.0 java-1.8.0-openjdk-headless-debug-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-devel-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-src-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-debug-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-src-debug-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-debuginfo-1.8.0.71-2.b15.el7_2.x86_64 java-1.8.0-openjdk-headless-1.8.0.71-2.b15.el7_2.x86_64 Steps to Reproduce: We do not have steps to reproduce yet. The code basically starts Eclipse without workbench (UI) and runs Java and Xtext based compiler over few files. Actual results: The execution terminated after few seconds with the error below: *** Error in `/usr/lib/jvm/java-1.8.0/bin/java': double free or corruption (out): 0x00000007c0036fc0 *** # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fe5aa133e75, pid=11123, tid=140622636713728 # # JRE version: OpenJDK Runtime Environment (8.0_71-b15) (build 1.8.0_71-b15) # Java VM: OpenJDK 64-Bit Server VM (25.71-b15 mixed mode linux-amd64 compressed oops) # Problematic frame: # V======= Backtrace: ========= [libjvm.so+0x5bae75]/lib64/libc.so.6(+0x7d023)[0x7fe5aaaf1023] Expected results: No crash Additional info: I'm attaching the *incompleted* crash file written by the JVM and the output printed to the console at same time.