Description of problem: At seemingly random times, I get an RHQ server crash in Fedora 13, with manually installed Sun JDK 1.6.0_20, glibc version is 2.12-1. uname -a gives "Linux localhost.localdomain 2.6.33.5-112.fc13.i686.PAE #1 SMP Thu May 27 02:56:20 UTC 2010 i686 i686 i386 GNU/Linux" This is the glibc generated backtrace I get on the server output: *** glibc detected *** /home/metlos/Progs/java6/bin/java: invalid fastbin entry (free): 0x634bed88 *** ======= Backtrace: ========= /lib/libc.so.6[0xc191e1] /home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libzip.so(Java_java_util_zip_Inflater_inflateBytes+0x116)[0xb6cacaa6] [0xb3b86c77] [0xb3ba8c04] ======= Memory map: ======== 003ce000-003eb000 r-xp 00000000 fd:01 12008 /lib/libgcc_s-4.4.4-20100503.so.1 003eb000-003ec000 rwxp 0001d000 fd:01 12008 /lib/libgcc_s-4.4.4-20100503.so.1 00b88000-00ba6000 r-xp 00000000 fd:01 11714 /lib/ld-2.12.so 00ba6000-00ba7000 r-xp 0001d000 fd:01 11714 /lib/ld-2.12.so 00ba7000-00ba8000 rwxp 0001e000 fd:01 11714 /lib/ld-2.12.so 00baa000-00d30000 r-xp 00000000 fd:01 11715 /lib/libc-2.12.so 00d30000-00d32000 r-xp 00185000 fd:01 11715 /lib/libc-2.12.so 00d32000-00d33000 rwxp 00187000 fd:01 11715 /lib/libc-2.12.so 00d33000-00d36000 rwxp 00000000 00:00 0 00d38000-00d60000 r-xp 00000000 fd:01 11831 /lib/libm-2.12.so 00d60000-00d61000 r-xp 00027000 fd:01 11831 /lib/libm-2.12.so 00d61000-00d62000 rwxp 00028000 fd:01 11831 /lib/libm-2.12.so 00d64000-00d67000 r-xp 00000000 fd:01 11720 /lib/libdl-2.12.so 00d67000-00d68000 r-xp 00002000 fd:01 11720 /lib/libdl-2.12.so 00d68000-00d69000 rwxp 00003000 fd:01 11720 /lib/libdl-2.12.so 00d6b000-00d82000 r-xp 00000000 fd:01 11727 /lib/libpthread-2.12.so 00d82000-00d83000 r-xp 00016000 fd:01 11727 /lib/libpthread-2.12.so 00d83000-00d84000 rwxp 00017000 fd:01 11727 /lib/libpthread-2.12.so 00d84000-00d86000 rwxp 00000000 00:00 0 00d9d000-00da4000 r-xp 00000000 fd:01 11728 /lib/librt-2.12.so 00da4000-00da5000 r-xp 00006000 fd:01 11728 /lib/librt-2.12.so 00da5000-00da6000 rwxp 00007000 fd:01 11728 /lib/librt-2.12.so 07dc8000-07ddf000 r-xp 00000000 fd:01 17246 /lib/libnsl-2.12.so 07ddf000-07de0000 r-xp 00016000 fd:01 17246 /lib/libnsl-2.12.so 07de0000-07de1000 rwxp 00017000 fd:01 17246 /lib/libnsl-2.12.so 07de1000-07de3000 rwxp 00000000 00:00 0 08048000-08052000 r-xp 00000000 fd:01 2107316 /home/metlos/Progs/jdk1.6.0_20/bin/java 08052000-08053000 rwxp 00009000 fd:01 2107316 /home/metlos/Progs/jdk1.6.0_20/bin/java 080e1000-09b2d000 rwxp 00000000 00:00 0 [heap] 5f345000-5f348000 ---p 00000000 00:00 0 5f348000-5f396000 rwxp 00000000 00:00 0 5f396000-5f399000 ---p 00000000 00:00 0 5f399000-5f3e7000 rwxp 00000000 00:00 0 5f3e7000-5f3ea000 ---p 00000000 00:00 0 5f3ea000-5f438000 rwxp 00000000 00:00 0 5f438000-5f43b000 ---p 00000000 00:00 0 5f43b000-5f489000 rwxp 00000000 00:00 0 5f4c9000-5f4cc000 ---p 00000000 00:00 0 5f4cc000-5f51a000 rwxp 00000000 00:00 0 5f51a000-5f51d000 ---p 00000000 00:00 0 5f51d000-5f56b000 rwxp 00000000 00:00 0 5f56b000-5f56e000 ---p 00000000 00:00 0 5f56e000-5f5bc000 rwxp 00000000 00:00 0 5f5bc000-5f5bf000 ---p 00000000 00:00 0 5f5bf000-5f60d000 rwxp 00000000 00:00 0 5f60d000-5f610000 ---p 00000000 00:00 0 5f610000-5f65e000 rwxp 00000000 00:00 0 5f65e000-5f661000 ---p 00000000 00:00 0 5f661000-5f6af000 rwxp 00000000 00:00 0 5f6af000-5f6b2000 ---p 00000000 00:00 0 5f6b2000-5f700000 rwxp 00000000 00:00 0 5f700000-5f7be000 rwxp 00000000 00:00 0 5f7be000-5f800000 ---p 00000000 00:00 0 5f80d000-5f810000 ---p 00000000 00:00 0 5f810000-5f85e000 rwxp 00000000 00:00 0 5f85e000-5f861000 ---p 00000000 00:00 0 5f861000-5f8af000 rwxp 00000000 00:00 0 5f8af000-5f8b2000 ---p 00000000 00:00 0 5f8b2000-5f900000 rwxp 00000000 00:00 0 5f900000-5fa00000 rwxp 00000000 00:00 0 5fa00000-5fafc000 rwxp 00000000 00:00 0 5fafc000-5fb00000 ---p 00000000 00:00 0 5fb0d000-5fb10000 ---p 00000000 00:00 0 5fb10000-5fb5e000 rwxp 00000000 00:00 0 5fb5e000-5fb61000 ---p 00000000 00:00 0 5fb61000-5fbaf000 rwxp 00000000 00:00 0 5fbaf000-5fbb2000 ---p 00000000 00:00 0 5fbb2000-5fc00000 rwxp 00000000 00:00 0 5fc00000-5fd00000 rwxp 00000000 00:00 0 5fd42000-5fd45000 ---p 00000000 00:00 0 5fd45000-5fd93000 rwxp 00000000 00:00 0 5fd93000-5fd96000 ---p 00000000 00:00 0 5fd96000-5fde4000 rwxp 00000000 00:00 0 5fdf9000-5fe0d000 r-xs 000f2000 fd:01 2650584 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/org.jboss.seam-jboss-seam-2.1.0.SP1.jar 5fe0d000-5fe91000 r-xp 00000000 fd:01 2104744 /home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libawt.so 5fe91000-5fe98000 rwxp 00084000 fd:01 2104744 /home/metlos/Progs/jdk1.6.0_20/jre/lib/i386/libawt.so 5fe98000-5febc000 rwxp 00000000 00:00 0 5febc000-5febf000 ---p 00000000 00:00 0 5febf000-5ff0d000 rwxp 00000000 00:00 0 5ff0d000-5ff10000 ---p 00000000 00:00 0 5ff10000-5ff5e000 rwxp 00000000 00:00 0 5ff5e000-5ff61000 ---p 00000000 00:00 0 5ff61000-5ffaf000 rwxp 00000000 00:00 0 5ffaf000-5ffb2000 ---p 00000000 00:00 0 5ffb2000-60000000 rwxp 00000000 00:00 0 60000000-600fa000 rwxp 00000000 00:00 0 600fa000-60100000 ---p 00000000 00:00 0 60114000-60119000 r-xs 0003b000 fd:01 2650586 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/jboss-seam-ui-2.1.0.SP1.jar 6011c000-6011f000 ---p 00000000 00:00 0 6011f000-6016d000 rwxp 00000000 00:00 0 60192000-601a7000 r-xs 0016a000 fd:01 2650591 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/richfaces-impl-3.3.3.CR1.jar 601a7000-601ad000 r-xs 00044000 fd:01 2650589 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/jsf-facelets-1.1.15.B1.jar 601ad000-601b2000 r-xs 00016000 fd:01 2650562 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/xml-apis-1.0.b2.jar 601b2000-601b4000 r-xs 0000d000 fd:01 2650587 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/struts-menu-2.3.jar 601b4000-60200000 r-xs 003f0000 fd:01 2650567 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/server/default/deploy/rhq.ear/rhq-portal.war/WEB-INF/lib/richfaces-ui-3.3.3.CR1.jar 60200000-6022e000 rwxp 00000000 00:00 0 6022e000-60300000 ---p 00000000 00:00 0 60300000-603f8000 rwxp 00000000 00:00 0 603f8000-60400000 ---p 00000000 00:00 0 60400000-604fc000 rwxp 00000000 00:00 0 604fc000-60500000 ---p 00000000 00:00 0 60500000-605fa000 rwxp 00000000 00:00 0 605fa000-60600000 ---p 00000000 00:00 0 60600000-606f9000 rwxp 00000000 00:00 0 606f9000-60700000 ---p 00000000 00:00 0 60700000-607fe000 rwxp 00000000 00:00 0 607fe000-60800000 ---p 00000000 00:00 0 60800000-608fa000 rwxp 00000000 00:00 0 608fa000-60900000 ---p 00000000 00:00 0 /home/metlos/Projects/java/jon/qa/jon-server-2.4.0.Beta1/server/jbossas/bin/run.sh: line 257: 4541 Aborted (core dumped) "$JAVA" $JAVA_OPTS -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" -classpath "$JBOSS_CLASSPATH" org.jboss.Main "$@" How reproducible: This seems to be happening at random times with no apparent action that would trigger this behaviour consistently. However it seems to be happening more frequently during heavy class-loading activities like redeploying a running application or during JON installation when the rhq.ear is being deployed for the first time.
This has just happened for me while starting up a regular JBoss AS without any RHQ bits installed. It seems to be an issue with either the system libs or the java version (or both) that I am running.
whoops, disregard the previous comment. The JBoss AS start up failure was different from the reported issue.
If anyone gets clear reproduction steps or sees this on one of JON's supported OSs then please set its priority to Urgent
Dropping severity until able to reproduce
Triaged 21-Sept
Tested on RHQ-Master build #517 QE environment: F13 (2.6.33.3-85.fc13.x86_64 #1 SMP Thu May 6 18:09:49 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux) glibc-2.12.1-2.x86_64 SUN java version "1.6.0_18" Postgres 8.4 Observation: Unable to reproduce server crash on F13.
RHQ server (RHQ-Master build #526) working fine on F13.