Bug 976664

Summary:	JVM crash (7.0_17-b02) in JDG 6.1.0.GA-redhat-2
Product:	[JBoss] JBoss Data Grid 6	Reporter:	Radim Vansa <rvansa>
Component:	Server	Assignee:	Tristan Tarrant <ttarrant>
Status:	CLOSED CANTFIX	QA Contact:	Martin Gencur <mgencur>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	unspecified	CC:	jdg-bugs
Target Milestone:	---
Target Release:	One-off release
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-06-26 09:41:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	975081

Description Radim Vansa 2013-06-21 06:53:34 UTC

JVM crashes of the same type occurred multiple times during stress-tests.

The crash log below can be found on https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-perf-cs-leveldb/ws/hs_err_pid24541.log
Core dump is located on https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-perf-cs-leveldb/ws/core.24541

More crash logs along with core dumps can be found in
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-perf-cs-leveldb/ws/

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fe66e041d37, pid=24541, tid=140622771934976
#
# JRE version: 7.0_17-b02
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: /qa/hudson_workspace/workspace/jdg-perf-cs-leveldb/core or core.24541
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007fe5c804c800):  JavaThread "leveldb-compaction-0" [_thread_in_Java, id=25036, stack(0x00007fe54a4e5000,0x00007fe54a5e6000)]

siginfo:si_signo=SIGBUS: si_errno=0, si_code=2 (BUS_ADRERR), si_addr=0x00007fe5ce919000

Registers:
RAX=0x00000000adfa3d46, RBX=0x0000000000000017, RCX=0x0000000000000007, RDX=0x0000000000000000
RSP=0x00007fe54a5e4750, RBP=0x00007fe54a5e4750, RSI=0x00007fe5ce918ff8, RDI=0x00000006f37863f8
R8 =0x000000071a0e8498, R9 =0x0000000000018000, R10=0x00007fe66e042760, R11=0x00007fe5ce919000
R12=0x0000000000000000, R13=0x00000006f37863f0, R14=0x00007fe5ce919007, R15=0x00007fe5c804c800
RIP=0x00007fe66e041d37, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007fe54a5e4750)
0x00007fe54a5e4750:   0000000000000000 00007fe66ec49be7
0x00007fe54a5e4760:   0000000720661610 0000000000000007
0x00007fe54a5e4770:   000000071a0e8498 0000000700018000
0x00007fe54a5e4780:   0000000720661610 0000018c00000007
0x00007fe54a5e4790:   0000000720d65810 00000006f37863d0
0x00007fe54a5e47a0:   de6f0c7e00000001 00000006f3786318
0x00007fe54a5e47b0:   00000006f37863a0 0000005700000007
0x00007fe54a5e47c0:   ffffff9dffffff93 0000018300000183
0x00007fe54a5e47d0:   fffffff8ffffffdb bb684693ffffffba
0x00007fe54a5e47e0:   00000037ffffffaf 00000006f3785d68
0x00007fe54a5e47f0:   0000000000000031 0000000000000246
0x00007fe54a5e4800:   00000006f3784c50 00007fe66f38329c
0x00007fe54a5e4810:   00000006f36991d8 0000018cde6d31e7
0x00007fe54a5e4820:   00000006f3698f38 00000006f3698f18
0x00007fe54a5e4830:   de6d323be40e3eec 000000072071f760
0x00007fe54a5e4840:   0000000000000000 0000000720134460
0x00007fe54a5e4850:   00000006f3699578 de6f098ae402688c
0x00007fe54a5e4860:   00000006f3786318 00000006f3785c80
0x00007fe54a5e4870:   e40cc2c800000081 0000000719f3acb8
0x00007fe54a5e4880:   0000000720cc0c80 00000006f3699170
0x00007fe54a5e4890:   0000000720a85968 00007fe66e1345b4
0x00007fe54a5e48a0:   00000000e40cc2b6 00007fe66ed54a60
0x00007fe54a5e48b0:   0000000000000007 00000007e2da26b8
0x00007fe54a5e48c0:   00000005cffbe801 0000000000000001
0x00007fe54a5e48d0:   00007fe54a5e4ae0 00007fe66e7294d4
0x00007fe54a5e48e0:   00000000000000c4 00000007e0aa1488
0x00007fe54a5e48f0:   0000000000000000 0000000000000000
0x00007fe54a5e4900:   0000000000000000 0000000000000000
0x00007fe54a5e4910:   0000000000000000 0000000000000000
0x00007fe54a5e4920:   0000000000000000 0000000000000000
0x00007fe54a5e4930:   0000000000000000 0000000000000000
0x00007fe54a5e4940:   0000000000000000 0000000000000000 

Instructions: (pc=0x00007fe66e041d37)
0x00007fe66e041d17:   da e9 79 00 00 00 48 8b 44 d7 08 48 89 44 d6 08
0x00007fe66e041d27:   48 ff c2 75 f1 f7 c1 04 00 00 00 74 0e 8b 47 08
0x00007fe66e041d37:   89 46 08 48 83 c7 04 48 83 c6 04 f7 c1 02 00 00
0x00007fe66e041d47:   00 74 10 66 8b 47 08 66 89 46 08 48 83 c7 02 48 

Register to memory mapping:

RAX=0x00000000adfa3d46 is an unknown value
RBX=0x0000000000000017 is an unknown value
RCX=0x0000000000000007 is an unknown value
RDX=0x0000000000000000 is an unknown value
RSP=0x00007fe54a5e4750 is pointing into the stack for thread: 0x00007fe5c804c800
RBP=0x00007fe54a5e4750 is pointing into the stack for thread: 0x00007fe5c804c800
RSI=0x00007fe5ce918ff8 is an unknown value
RDI=

Comment 1 Radim Vansa 2013-06-21 06:55:09 UTC

I'll check if this occurs on other JVMs (1.6)

Comment 2 Radim Vansa 2013-06-21 09:43:24 UTC

The same error occurred on all machines with JVM versions 6.0_43-b01 and 7.0_09-b05

Comment 3 Radim Vansa 2013-06-24 13:19:05 UTC

LevelDB uses reflection in order to unmap memory-mapped files in an unsafe way, according to http://bugs.sun.com/view_bug.do?bug_id=4724038
This may be reason why the JVM is crashing.

Suggested workaround is command-line flag -Dleveldb.mmap=false which forces using FileChannelTable implementation for accessing the disk.

Comment 4 Tristan Tarrant 2013-06-26 09:41:30 UTC

Since this is a JVM bug, and there is a workaround I'm closing this issue as CANTFIX.