Bug 601117

Summary:	Server crash on F13
Product:	[Other] RHQ Project	Reporter:	Lukas Krejci <lkrejci>
Component:	Usability	Assignee:	RHQ Project Maintainer <rhq-maint>
Status:	CLOSED WORKSFORME	QA Contact:	Corey Welton <cwelton>
Severity:	medium	Docs Contact:
Priority:	high
Version:	3.0.0	CC:	rtimaniy
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-10-29 08:51:58 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	591531, 593121

Description Lukas Krejci 2010-06-07 09:31:10 UTC

Description of problem:

At seemingly random times,  I get an RHQ server crash in Fedora 13, with manually installed Sun JDK 1.6.0_20, glibc version is 2.12-1.
uname -a gives "Linux localhost.localdomain 2.6.33.5-112.fc13.i686.PAE #1 SMP Thu May 27 02:56:20 UTC 2010 i686 i686 i386 GNU/Linux"

This is the glibc generated backtrace I get on the server output:

*** glibc detected *** /home/metlos/Progs/java6/bin/java: invalid fastbin entry (free): 0x634bed88 ***
======= Backtrace: =========
/lib/libc.so.6[0xc191e1]
/home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libzip.so(Java_java_util_zip_Inflater_inflateBytes+0x116)[0xb6cacaa6]
[0xb3b86c77]
[0xb3ba8c04]
======= Memory map: ========
003ce000-003eb000 r-xp 00000000 fd:01 12008      /lib/libgcc_s-4.4.4-20100503.so.1
003eb000-003ec000 rwxp 0001d000 fd:01 12008      /lib/libgcc_s-4.4.4-20100503.so.1
00b88000-00ba6000 r-xp 00000000 fd:01 11714      /lib/ld-2.12.so
00ba6000-00ba7000 r-xp 0001d000 fd:01 11714      /lib/ld-2.12.so
00ba7000-00ba8000 rwxp 0001e000 fd:01 11714      /lib/ld-2.12.so
00baa000-00d30000 r-xp 00000000 fd:01 11715      /lib/libc-2.12.so
00d30000-00d32000 r-xp 00185000 fd:01 11715      /lib/libc-2.12.so
00d32000-00d33000 rwxp 00187000 fd:01 11715      /lib/libc-2.12.so
00d33000-00d36000 rwxp 00000000 00:00 0 
00d38000-00d60000 r-xp 00000000 fd:01 11831      /lib/libm-2.12.so
00d60000-00d61000 r-xp 00027000 fd:01 11831      /lib/libm-2.12.so
00d61000-00d62000 rwxp 00028000 fd:01 11831      /lib/libm-2.12.so
00d64000-00d67000 r-xp 00000000 fd:01 11720      /lib/libdl-2.12.so
00d67000-00d68000 r-xp 00002000 fd:01 11720      /lib/libdl-2.12.so
00d68000-00d69000 rwxp 00003000 fd:01 11720      /lib/libdl-2.12.so
00d6b000-00d82000 r-xp 00000000 fd:01 11727      /lib/libpthread-2.12.so
00d82000-00d83000 r-xp 00016000 fd:01 11727      /lib/libpthread-2.12.so
00d83000-00d84000 rwxp 00017000 fd:01 11727      /lib/libpthread-2.12.so
00d84000-00d86000 rwxp 00000000 00:00 0 
00d9d000-00da4000 r-xp 00000000 fd:01 11728      /lib/librt-2.12.so
00da4000-00da5000 r-xp 00006000 fd:01 11728      /lib/librt-2.12.so
00da5000-00da6000 rwxp 00007000 fd:01 11728      /lib/librt-2.12.so
07dc8000-07ddf000 r-xp 00000000 fd:01 17246      /lib/libnsl-2.12.so
07ddf000-07de0000 r-xp 00016000 fd:01 17246      /lib/libnsl-2.12.so
07de0000-07de1000 rwxp 00017000 fd:01 17246      /lib/libnsl-2.12.so
07de1000-07de3000 rwxp 00000000 00:00 0 
08048000-08052000 r-xp 00000000 fd:01 2107316    /home/metlos/Progs/jdk1.6.0_20/bin/java
08052000-08053000 rwxp 00009000 fd:01 2107316    /home/metlos/Progs/jdk1.6.0_20/bin/java
080e1000-09b2d000 rwxp 00000000 00:00 0          [heap]
5f345000-5f348000 ---p 00000000 00:00 0 
5f348000-5f396000 rwxp 00000000 00:00 0 
5f396000-5f399000 ---p 00000000 00:00 0 
5f399000-5f3e7000 rwxp 00000000 00:00 0 
5f3e7000-5f3ea000 ---p 00000000 00:00 0 
5f3ea000-5f438000 rwxp 00000000 00:00 0 
5f438000-5f43b000 ---p 00000000 00:00 0 
5f43b000-5f489000 rwxp 00000000 00:00 0 
5f4c9000-5f4cc000 ---p 00000000 00:00 0 
5f4cc000-5f51a000 rwxp 00000000 00:00 0 
5f51a000-5f51d000 ---p 00000000 00:00 0 
5f51d000-5f56b000 rwxp 00000000 00:00 0 
5f56b000-5f56e000 ---p 00000000 00:00 0 
5f56e000-5f5bc000 rwxp 00000000 00:00 0 
5f5bc000-5f5bf000 ---p 00000000 00:00 0 
5f5bf000-5f60d000 rwxp 00000000 00:00 0 
5f60d000-5f610000 ---p 00000000 00:00 0 
5f610000-5f65e000 rwxp 00000000 00:00 0 
5f65e000-5f661000 ---p 00000000 00:00 0 
5f661000-5f6af000 rwxp 00000000 00:00 0 
5f6af000-5f6b2000 ---p 00000000 00:00 0 
5f6b2000-5f700000 rwxp 00000000 00:00 0 
5f700000-5f7be000 rwxp 00000000 00:00 0 
5f7be000-5f800000 ---p 00000000 00:00 0 
5f80d000-5f810000 ---p 00000000 00:00 0 
5f810000-5f85e000 rwxp 00000000 00:00 0 
5f85e000-5f861000 ---p 00000000 00:00 0 
5f861000-5f8af000 rwxp 00000000 00:00 0 
5f8af000-5f8b2000 ---p 00000000 00:00 0 
5f8b2000-5f900000 rwxp 00000000 00:00 0 
5f900000-5fa00000 rwxp 00000000 00:00 0 
5fa00000-5fafc000 rwxp 00000000 00:00 0 
5fafc000-5fb00000 ---p 00000000 00:00 0 
5fb0d000-5fb10000 ---p 00000000 00:00 0 
5fb10000-5fb5e000 rwxp 00000000 00:00 0 
5fb5e000-5fb61000 ---p 00000000 00:00 0 
5fb61000-5fbaf000 rwxp 00000000 00:00 0 
5fbaf000-5fbb2000 ---p 00000000 00:00 0 
5fbb2000-5fc00000 rwxp 00000000 00:00 0 
5fc00000-5fd00000 rwxp 00000000 00:00 0 
5fd42000-5fd45000 ---p 00000000 00:00 0 
5fd45000-5fd93000 rwxp 00000000 00:00 0 
5fd93000-5fd96000 ---p 00000000 00:00 0 
5fd96000-5fde4000 rwxp 00000000 00:00 0 
5fdf9000-5fe0d000 r-xs 000f2000 fd:01 2650584    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/org.jboss.seam-jboss-seam-2.1.0.SP1.jar
5fe0d000-5fe91000 r-xp 00000000 fd:01 2104744    /home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libawt.so
5fe91000-5fe98000 rwxp 00084000 fd:01 2104744    /home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libawt.so
5fe98000-5febc000 rwxp 00000000 00:00 0 
5febc000-5febf000 ---p 00000000 00:00 0 
5febf000-5ff0d000 rwxp 00000000 00:00 0 
5ff0d000-5ff10000 ---p 00000000 00:00 0 
5ff10000-5ff5e000 rwxp 00000000 00:00 0 
5ff5e000-5ff61000 ---p 00000000 00:00 0 
5ff61000-5ffaf000 rwxp 00000000 00:00 0 
5ffaf000-5ffb2000 ---p 00000000 00:00 0 
5ffb2000-60000000 rwxp 00000000 00:00 0 
60000000-600fa000 rwxp 00000000 00:00 0 
600fa000-60100000 ---p 00000000 00:00 0 
60114000-60119000 r-xs 0003b000 fd:01 2650586    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/jboss-seam-ui-2.1.0.SP1.jar
6011c000-6011f000 ---p 00000000 00:00 0 
6011f000-6016d000 rwxp 00000000 00:00 0 
60192000-601a7000 r-xs 0016a000 fd:01 2650591    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/richfaces-impl-3.3.3.CR1.jar
601a7000-601ad000 r-xs 00044000 fd:01 2650589    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/jsf-facelets-1.1.15.B1.jar
601ad000-601b2000 r-xs 00016000 fd:01 2650562    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/xml-apis-1.0.b2.jar
601b2000-601b4000 r-xs 0000d000 fd:01 2650587    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/struts-menu-2.3.jar
601b4000-60200000 r-xs 003f0000 fd:01 2650567    /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/richfaces-ui-3.3.3.CR1.jar
60200000-6022e000 rwxp 00000000 00:00 0 
6022e000-60300000 ---p 00000000 00:00 0 
60300000-603f8000 rwxp 00000000 00:00 0 
603f8000-60400000 ---p 00000000 00:00 0 
60400000-604fc000 rwxp 00000000 00:00 0 
604fc000-60500000 ---p 00000000 00:00 0 
60500000-605fa000 rwxp 00000000 00:00 0 
605fa000-60600000 ---p 00000000 00:00 0 
60600000-606f9000 rwxp 00000000 00:00 0 
606f9000-60700000 ---p 00000000 00:00 0 
60700000-607fe000 rwxp 00000000 00:00 0 
607fe000-60800000 ---p 00000000 00:00 0 
60800000-608fa000 rwxp 00000000 00:00 0 
608fa000-60900000 ---p 00000000 00:00 0 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/bin/run.sh: line 257:  4541 Aborted                 (core dumped) "$JAVA" $JAVA_OPTS -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" -classpath "$JBOSS_CLASSPATH" org.jboss.Main "$@"

How reproducible:
This seems to be happening at random times with no apparent action that would trigger this behaviour consistently. However it seems to be happening more frequently during heavy class-loading activities like redeploying a running application or during JON installation when the rhq.ear is being deployed for the first time.

Comment 1 Lukas Krejci 2010-06-07 13:06:14 UTC

This has just happened for me while starting up a regular JBoss AS without any RHQ bits installed.

It seems to be an issue with either the system libs or the java version (or both) that I am running.

Comment 2 Lukas Krejci 2010-06-07 13:08:44 UTC

whoops, disregard the previous comment. The JBoss AS start up failure was different from the reported issue.

Comment 3 Charles Crouch 2010-06-07 13:18:24 UTC

If anyone gets clear reproduction steps or sees this on one of JON's supported OSs then please set its priority to Urgent

Comment 4 Charles Crouch 2010-07-20 14:55:28 UTC

Dropping severity until able to reproduce

Comment 5 Corey Welton 2010-09-24 12:54:54 UTC

Triaged 21-Sept

Comment 6 Rajan Timaniya 2010-10-28 10:13:09 UTC

Tested on RHQ-Master build #517

QE environment:
F13 (2.6.33.3-85.fc13.x86_64 #1 SMP Thu May 6 18:09:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux)
glibc-2.12.1-2.x86_64
SUN java version "1.6.0_18"
Postgres 8.4

Observation:
Unable to reproduce server crash on F13.

Comment 7 Rajan Timaniya 2010-10-29 08:51:58 UTC

RHQ server (RHQ-Master build #526) working fine on F13.