Bug 88399 - segfault in /lib/libpthread.so.0
segfault in /lib/libpthread.so.0
Status: CLOSED CURRENTRELEASE
Product: Red Hat Linux
Classification: Retired
Component: glibc (Show other bugs)
8.0
athlon Linux
medium Severity high
: ---
: ---
Assigned To: Jakub Jelinek
Brian Brock
http://p1.selectacast.net/~jks/hs_err...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2003-04-09 15:38 EDT by Joseph Shraibman
Modified: 2007-04-18 12:52 EDT (History)
0 users

See Also:
Fixed In Version: 2.3.2-4.80.6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2003-10-08 18:57:17 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Joseph Shraibman 2003-04-09 15:38:56 EDT
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4a) Gecko/20030401

Description of problem:
When running a java program that uses a lot of threads (up to 3000ish) I get this:
An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x4002CFEC
Function=(null)+0x4002CFEC
Library=/lib/libpthread.so.0

NOTE: We are unable to locate the function name symbol for the error
      just occurred. Please refer to release documentation for possible
      reason and solutions.


Current Java thread:
	at java.lang.Thread.start(Native Method)
	- locked <0x4578a288> (a java.lang.Thread)
	at org.xbill.DNS.SimpleResolver.sendAsync(SimpleResolver.java:315)
	at org.xbill.DNS.ExtendedResolver.sendTo(ExtendedResolver.java:148)
	- locked <0x45789ed0> (a java.util.HashMap)
	at org.xbill.DNS.ExtendedResolver.send(ExtendedResolver.java:248)
	- locked <0x45789ea0> (a java.util.LinkedList)
	at org.xbill.DNS.Lookup.lookup(Lookup.java:384)
	at org.xbill.DNS.Lookup.resolve(Lookup.java:424)
	at org.xbill.DNS.Lookup.run(Lookup.java:441)
	at org.xbill.DNS.dns.getRecords(dns.java:146)
	at org.xbill.DNS.dns.getAnyRecords(dns.java:195)
        <snip>


Dynamic libraries:
08048000-0804e000 r-xp 00000000 03:03 2245061    /usr/local/j2sdk1.4.1_02/bin/java
0804e000-0804f000 rw-p 00005000 03:03 2245061    /usr/local/j2sdk1.4.1_02/bin/java
40000000-40012000 r-xp 00000000 03:03 376907     /lib/ld-2.3.2.so
40012000-40013000 rw-p 00011000 03:03 376907     /lib/ld-2.3.2.so
40013000-4001c000 r-xp 00000000 03:03 1655143   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/native_threads/libhpi.so
4001c000-4001d000 rw-p 00008000 03:03 1655143   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/native_threads/libhpi.so
4001e000-4001f000 r--p 00a08000 03:03 426253     /usr/lib/locale/locale-archive
4001f000-40022000 r--s 00000000 03:03 2457778   
/usr/local/j2sdk1.4.1_02/jre/lib/ext/dnsns.jar
40022000-40030000 r-xp 00000000 03:03 376926     /lib/libpthread-0.10.so
40030000-40033000 rw-p 0000e000 03:03 376926     /lib/libpthread-0.10.so
40074000-40076000 r-xp 00000000 03:03 376857     /lib/libdl-2.3.2.so
40076000-40077000 rw-p 00001000 03:03 376857     /lib/libdl-2.3.2.so
40077000-40199000 r-xp 00000000 03:03 376854     /lib/libc-2.3.2.so
40199000-4019e000 rw-p 00121000 03:03 376854     /lib/libc-2.3.2.so
401a1000-4063a000 r-xp 00000000 03:03 2097204   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/server/libjvm.so
4063a000-40872000 rw-p 00498000 03:03 2097204   
/usr/local/j2sdk1.4.1_02/jre/lib/i386/server/libjvm.so
<rest snipped becuse if I don't apache complains that the request uri is too long>

After that error appears the jvm doesn't core dump or go away, it just sits
there.  Existing rmi calls just never return and new ones result in:
java.rmi.ConnectIOException: error during JRMP connection establishment; nested
exception is: java.net.SocketTimeoutException: Read timed out

Version-Release number of selected component (if applicable):
glibc-2.3.2-4.80

How reproducible:
Sometimes

Steps to Reproduce:
Seems hard to reproduce.  It seems to require a lot of threads and high memory usage

    

Additional info:

redhat 8.0 dual athlon system
java version 1.4.1_02 from java.sun.com

We never got this crash before the recent glibc upgrade, but that doesn't mean
anything. This crash also occured with jvm 1.4.1_01. The crash occurs when
spinning off a thread. We have had this crash with in different parts of the
code that spin off threads.  It occurs with client and server jvms.
Comment 1 Jakub Jelinek 2003-04-09 15:46:13 EDT
Please try ftp://people.redhat.com/jakub/glibc/errata/8.0/
2.3.2-4.80 because of a kernel bug doesn't use /lib/i686/libpthread.so.0
but /lib/libpthread.so.0 (that is a compatibility with old kernels version,
with fixed size 2MB thread stacks, so with 3000 * 2M you might very well
reach address space limits (libraries are mapped at around 1GB, kernel starts
at 3GB and thread stack array is in between).
Comment 2 Joseph Shraibman 2003-04-09 16:18:42 EDT
What is the kernel bug?  Will I run into it if I install these rpms?  Do I need
to reboot my machine after this?
Comment 3 Joseph Shraibman 2003-04-09 22:40:17 EDT
And is this fixed in redhat 9?
Comment 4 Jakub Jelinek 2003-04-10 18:44:56 EDT
The kernel bug is that AT_PLATFORM aux vector is passed wrong value to the
applications by the kernel. Try LD_SHOW_AUXV=1 /bin/echo, AT_PLATFORM
should be i686 on i686+ box, but 8.0 kernel passes an empty string.
This is fixed in RHL9 kernel and hopefully will be fixed in next 8.0 errata
kernel too when/if that happens.
As for the glibc errata, 2.3.2-4.80-6 will be pushed officially as errata
today.
Comment 5 Joseph Shraibman 2003-04-14 14:42:59 EDT
Upgrading to glibc-2.3.2-4.80.6 solved this problem.
Comment 6 Joseph Shraibman 2003-10-07 23:55:57 EDT
I just upgraded a system to redhat 9 and this bug seems to have regressed.

An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : 11 occurred at PC=0x4002CB28
Function=(null)+0x4002CB28
Library=/lib/libpthread.so.0

NOTE: We are unable to locate the function name symbol for the error
      just occurred. Please refer to release documentation for possible
      reason and solutions.


Current Java thread:
        at java.lang.Thread.start(Native Method)
        - locked <0x44cd3df8> (a java.lang.Thread)
        at com.xtenit.mail.DeliverySystem2Imp.deliver0(DeliverySystem2Imp.java:923)
        at
com.xtenit.mail.DeliverySystem2Imp.deliver0wrapper(DeliverySystem2Imp.java:496)
        at com.xtenit.mail.DeliverySystem2Imp.deliver(DeliverySystem2Imp.java:481)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
        at sun.rmi.transport.Transport$1.run(Transport.java:148)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
        at java.lang.Thread.run(Thread.java:534)

Dynamic libraries:
08048000-0804e000 r-xp 00000000 03:03 2736898    /usr/local/j2sdk1.4.2/bin/java
0804e000-0804f000 rw-p 00005000 03:03 2736898    /usr/local/j2sdk1.4.2/bin/java
40000000-40012000 r-xp 00000000 03:03 376922     /lib/ld-2.3.2.so
40012000-40013000 rw-p 00011000 03:03 376922     /lib/ld-2.3.2.so
40014000-4001c000 r-xp 00000000 03:03 1294916   
/usr/local/j2sdk1.4.2/jre/lib/i386/native_threads/libhpi.so
4001c000-4001d000 rw-p 00007000 03:03 1294916   
/usr/local/j2sdk1.4.2/jre/lib/i386/native_threads/libhpi.so
4001d000-40021000 rw-s 00000000 03:03 868361     /tmp/hsperfdata_runner/22448
40021000-40022000 r--p 00a08000 03:03 425990     /usr/lib/locale/locale-archive
40022000-40030000 r-xp 00000000 03:03 376917     /lib/libpthread-0.10.so
40030000-40033000 rw-p 0000e000 03:03 376917     /lib/libpthread-0.10.so
40073000-40075000 r-xp 00000000 03:03 376860     /lib/libdl-2.3.2.so
40075000-40076000 rw-p 00002000 03:03 376860     /lib/libdl-2.3.2.so
40076000-40196000 r-xp 00000000 03:03 376837     /lib/libc-2.3.2.so
40196000-40199000 rw-p 00120000 03:03 376837     /lib/libc-2.3.2.so
4019c000-40748000 r-xp 00000000 03:03 1294918   
/usr/local/j2sdk1.4.2/jre/lib/i386/server/libjvm.so
40748000-407a2000 rw-p 005ab000 03:03 1294918   
/usr/local/j2sdk1.4.2/jre/lib/i386/server/libjvm.so
407b5000-407c5000 r-xp 00000000 03:03 376868     /lib/libnsl-2.3.2.so
407c5000-407c6000 rw-p 00010000 03:03 376868     /lib/libnsl-2.3.2.so
407c8000-407e8000 r-xp 00000000 03:03 376866     /lib/libm-2.3.2.so
407e8000-407e9000 rw-p 00020000 03:03 376866     /lib/libm-2.3.2.so
407e9000-407ec000 r--s 00000000 03:03 3801317   
/usr/local/j2sdk1.4.2/jre/lib/ext/dnsns.jar
407ec000-407f4000 r-xp 00000000 03:03 376907     /lib/libnss_nis-2.3.2.so
407f4000-407f5000 rw-p 00007000 03:03 376907     /lib/libnss_nis-2.3.2.so
407f5000-407f6000 r-xp 00000000 03:03 1098823   
/usr/local/j2sdk1.4.2/jre/lib/i386/librmi.so
407f6000-407f7000 rw-p 00000000 03:03 1098823   
/usr/local/j2sdk1.4.2/jre/lib/i386/librmi.so
407f7000-40801000 r-xp 00000000 03:03 376893     /lib/libnss_files-2.3.2.so
40801000-40802000 rw-p 00009000 03:03 376893     /lib/libnss_files-2.3.2.so
40802000-40812000 r-xp 00000000 03:03 1098805   
/usr/local/j2sdk1.4.2/jre/lib/i386/libverify.so
40812000-40814000 rw-p 0000f000 03:03 1098805   
/usr/local/j2sdk1.4.2/jre/lib/i386/libverify.so
40814000-40834000 r-xp 00000000 03:03 1098806   
/usr/local/j2sdk1.4.2/jre/lib/i386/libjava.so
40834000-40836000 rw-p 0001f000 03:03 1098806   
/usr/local/j2sdk1.4.2/jre/lib/i386/libjava.so
40836000-4084a000 r-xp 00000000 03:03 1098808   
/usr/local/j2sdk1.4.2/jre/lib/i386/libzip.so
4084a000-4084d000 rw-p 00013000 03:03 1098808   
/usr/local/j2sdk1.4.2/jre/lib/i386/libzip.so
4084d000-421d8000 r--s 00000000 03:03 1115216   
/usr/local/j2sdk1.4.2/jre/lib/rt.jar
42222000-42238000 r--s 00000000 03:03 1114890   
/usr/local/j2sdk1.4.2/jre/lib/sunrsasign.jar
42238000-42313000 r--s 00000000 03:03 1115213   
/usr/local/j2sdk1.4.2/jre/lib/jsse.jar
42313000-42324000 r--s 00000000 03:03 1114921   
/usr/local/j2sdk1.4.2/jre/lib/jce.jar
42324000-4287d000 r--s 00000000 03:03 1115214   
/usr/local/j2sdk1.4.2/jre/lib/charsets.jar
44925000-4492e000 r--s 00000000 03:03 1081522    /usr/local/java/classes/cos.jar
78c48000-78e48000 r--p 00000000 03:03 425990     /usr/lib/locale/locale-archive
78e48000-78e7a000 r--p 009ba000 03:03 425990     /usr/lib/locale/locale-archive
78e7a000-78e96000 r--s 00000000 03:03 3801316   
/usr/local/j2sdk1.4.2/jre/lib/ext/sunjce_provider.jar
78e96000-78ea3000 r--s 00000000 03:03 3801318   
/usr/local/j2sdk1.4.2/jre/lib/ext/ldapsec.jar
78ea3000-78f5f000 r--s 00000000 03:03 3801319   
/usr/local/j2sdk1.4.2/jre/lib/ext/localedata.jar
78f5f000-78faa000 r--s 00000000 03:03 1081481    /usr/local/java/classes/smtp.jar
78faa000-78fba000 r-xp 00000000 03:03 1098811   
/usr/local/j2sdk1.4.2/jre/lib/i386/libnet.so
78fba000-78fbb000 rw-p 0000f000 03:03 1098811   
/usr/local/j2sdk1.4.2/jre/lib/i386/libnet.so
78fbb000-78fc9000 r--s 00000000 03:03 1081451   
/usr/local/java/classes/activation.102.jar
78fc9000-78fe5000 r--s 00000000 03:03 1081774   
/usr/local/java/classes/xml-apis.xalan241.jar
78fe5000-78ffb000 r--s 00000000 03:03 1081514   
/usr/local/java/classes/NetComponents.jar
78ffb000-78ffe000 r-xp 00000000 03:03 376878     /lib/libnss_dns-2.3.2.so
78ffe000-78fff000 rw-p 00002000 03:03 376878     /lib/libnss_dns-2.3.2.so
79100000-79150000 r--s 00000000 03:03 1081779   
/usr/local/java/classes/mail-1.3.1ea.jar
79150000-7921c000 r--s 00000000 03:03 1081444   
/usr/local/java/classes/xerces.221.jar
7921c000-79318000 r--s 00000000 03:03 1081773   
/usr/local/java/classes/xalan.241.jar
79318000-79334000 r--s 00000000 03:03 1081774   
/usr/local/java/classes/xml-apis.xalan241.jar
79334000-79370000 r--s 00000000 03:03 1081780   
/usr/local/java/classes/bsh-1.3b2.jar
79370000-7937d000 r--s 00000000 03:03 1081546    /usr/local/java/classes/mbox.jar
7937d000-79385000 r--s 00000000 03:03 1081543   
/usr/local/java/classes/gnu-regexp-1.1.2.jar
79385000-793b3000 r--s 00000000 03:03 1081776   
/usr/local/java/classes/dnsjava-1.3.2.jar
793c1000-793ce000 r-xp 00000000 03:03 376926     /lib/libresolv-2.3.2.so
793ce000-793cf000 rw-p 0000d000 03:03 376926     /lib/libresolv-2.3.2.so

Heap at VM Abort:
Heap
 def new generation   total 8384K, used 5549K [0x44930000, 0x45240000, 0x49e80000)
  eden space 7488K,  62% used [0x44930000, 0x44dbb600, 0x45080000)
  from space 896K, 100% used [0x45080000, 0x45160000, 0x45160000)
  to   space 896K,   0% used [0x45160000, 0x45160000, 0x45240000)
 tenured generation   total 50556K, used 37999K [0x49e80000, 0x4cfdf000, 0x74930000)
   the space 50556K,  75% used [0x49e80000, 0x4c39bde0, 0x4c39be00, 0x4cfdf000)
 compacting perm gen  total 16384K, used 7293K [0x74930000, 0x75930000, 0x78930000)
   the space 16384K,  44% used [0x74930000, 0x7504f428, 0x7504f600, 0x75930000)

Local Time = Tue Oct  7 22:52:54 2003
Elapsed Time = 36848
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2-b28 mixed mode)
#
# An error report file has been saved as hs_err_pid22448.log.
# Please refer to the file for further information.
#


****************
Another exception has been detected while we were handling last error.
Dumping information about last error:
ERROR REPORT FILE = hs_err_pid22448.log
PC                = 0x4002cb28
SIGNAL            = 11
FUNCTION NAME     = (N/A)
OFFSET            = 0x4002CB28
LIBRARY NAME      = /lib/libpthread.so.0
Please check ERROR REPORT FILE for further information, if there is any.
Good bye.
Comment 7 Joseph Shraibman 2003-10-08 00:07:41 EDT
Also interesting:  On my single athlon laptop:
$ strace date 2>&1 | grep lib
open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
open("/lib/tls/librt.so.1", O_RDONLY)   = 3
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
$ uname -a
Linux jks-laptop 2.4.20-20.9 #1 Mon Aug 18 11:27:43 EDT 2003 i686 athlon i386
GNU/Linux

On my dual athlon server:
]# strace date 2>&1 | grep lib
open("/lib/libpthread.so.0", O_RDONLY)  = 3
open("/lib/librt.so.1", O_RDONLY)       = 3
open("/lib/libc.so.6", O_RDONLY)        = 3
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
]# uname -a
Linux p2.selectacast.net 2.4.20-20.9smp #1 SMP Mon Aug 18 11:18:01 EDT 2003 i686
athlon i386 GNU/Linux
Comment 8 Joseph Shraibman 2003-10-08 18:57:17 EDT
Never mind. It seems that during the upgrade somehow the i386 version of glibc
was installed instead of the i686 version.  So this isn't a glibc problem but a
different problem.

Perhaps a check can be added to the rpms to make sure that at least a warning is
printed if the i386 rpm gets installed on a better system?  Redhat won't run on
a 386 anymore anyway.

Note You need to log in before you can comment on or make changes to this bug.