Bug 1016359 - java abort in clear_hashtable due to invalid LVT_Hash*
java abort in clear_hashtable due to invalid LVT_Hash*
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: java-1.6.0-openjdk (Show other bugs)
5.5
Unspecified Unspecified
medium Severity medium
: rc
: 5.11
Assigned To: Andrew Haley
BaseOS QE - Apps
:
Depends On:
Blocks: 1049888
  Show dependency treegraph
 
Reported: 2013-10-07 22:51 EDT by Brad Hubbard
Modified: 2014-02-04 10:11 EST (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-28 12:16:38 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Full backtrace of all threads (13.45 KB, application/octet-stream)
2013-10-07 23:17 EDT, Brad Hubbard
no flags Details
jboss file being accessed at abort time (1.06 MB, application/octet-stream)
2013-10-07 23:18 EDT, Brad Hubbard
no flags Details
sosreport (2.07 MB, application/bzip2)
2013-10-07 23:19 EDT, Brad Hubbard
no flags Details

  None (edit)
Description Brad Hubbard 2013-10-07 22:51:57 EDT
Description of problem:

Core was generated by `java -Dprogram.name=run.sh -server -Xms1303m -Xmx1303m -XX:PermSize=256m -XX:Ma'.
Program terminated with signal 6, Aborted.

(gdb) bt
#0  0x000000380ba30265 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x000000380ba31d10 in abort () at abort.c:88
#2  0x000000380ba75e07 in munmap_chunk (mem=0x21bf0220, caller=<value optimized out>) at malloc.c:6210
#3  free_check (mem=0x21bf0220, caller=<value optimized out>) at hooks.c:288
#4  0x000000380ba727f1 in __libc_free (mem=0x6ed0) at malloc.c:3643
#5  0x00002b5eef2c1bf1 in clear_hashtable (table=0x25120140) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.cpp:1309
#6  0x00002b5eef2cabb0 in ClassFileParser::parse_method (this=0x42d93af0, cp=<value optimized out>, is_interface=<value optimized out>, promoted_flags=0x42d93a30, method_annotations=0x42d93710, method_parameter_annotations=0x42d93700, method_default_annotations=0x42d936f0, __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.cpp:2041
#7  0x00002b5eef2cb423 in ClassFileParser::parse_methods (this=0x42d93af0, cp=<value optimized out>, is_interface=<value optimized out>, promoted_flags=0x42d93a30, has_final_method=0x42d93a3f, methods_annotations_oop=0x42d939e8, methods_parameter_annotations_oop=0x42d939e0, methods_default_annotations_oop=0x42d939d8, __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.cpp:2117
#8  0x00002b5eef2cf1f8 in ClassFileParser::parseClassFile (this=0x42d93af0, name=<value optimized out>, class_loader=<value optimized out>, protection_domain=<value optimized out>, host_klass=<value optimized out>, cp_patches=<value optimized out>, parsed_name=..., verify=true, __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.cpp:3070
#9  0x00002b5eef728cc5 in parseClassFile (class_name=<value optimized out>, class_loader=<value optimized out>, protection_domain=..., st=0x42d93cc0, verify=true, __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.hpp:282
#10 SystemDictionary::resolve_from_stream (class_name=<value optimized out>, class_loader=<value optimized out>, protection_domain=..., st=0x42d93cc0, verify=true, __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp:1099
#11 0x00002b5eef4a9b83 in jvm_define_class_common (env=0x2293a9d0, name=0x2044e570 "PRk\"", loader=0x42d94360, buf=0x20f0c200 "\312\376\272\276", len=5158, pd=0x42d94338, source=0x42d93dd0 "jar:file:/opt/jboss-eap-5.1/jboss-as/common/lib/jboss-messaging.jar!/", verify=1 '\001', __the_thread__=0x2293a800) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/prims/jvm.cpp:864
#12 0x00002b5eef4b0ada in JVM_DefineClassWithSource (env=0x2293a9d0, name=0x42d941d0 "org/jboss/jms/message/BytesMessageProxy", loader=0x42d94360, buf=0x20f0c200 "\312\376\272\276", len=5158, pd=0x42d94338, source=0x42d93dd0 "jar:file:/opt/jboss-eap-5.1/jboss-as/common/lib/jboss-messaging.jar!/") at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/prims/jvm.cpp:884
#13 0x00002aaaaacdb48a in Java_java_lang_ClassLoader_defineClass1 (env=0x2293a9d0, loader=0x42d94360, name=0x42d94358, data=<value optimized out>, offset=<value optimized out>, length=5158, pd=0x42d94338, source=0x42d94330) at ../../../src/share/native/java/lang/ClassLoader.c:151
#14 0x00002aaaab32db06 in ?? ()
#15 0x0000000042d94338 in ?? ()
#16 0x0000000042d94330 in ?? ()
#17 0x0000000042d942a0 in ?? ()
#18 0x000000008ea29d21 in ?? ()
#19 0x0000000042d942f0 in ?? ()
#20 0x000000008ea2b178 in ?? ()
#21 0x000000009027d2c8 in ?? ()
#22 0x000000008ea29d38 in ?? ()
#23 0x0000000042d942d0 in ?? ()
#24 0x0000000000000000 in ?? ()

(gdb) f 5
#5  0x00002b5eef2c1bf1 in clear_hashtable (table=0x25120140) at /usr/src/debug/icedtea6-1.11.11.90/openjdk/hotspot/src/share/vm/classfile/classFileParser.cpp:1309
1309          delete(current);

(gdb) l clear_hashtable
1297      for (int i = 0; i < HASH_ROW_SIZE; i++) {
1298        table[i] = NULL;
1299      }
1300    }
1301
1302    void clear_hashtable(LVT_Hash** table) {
1303      for (int i = 0; i < HASH_ROW_SIZE; i++) {
1304        LVT_Hash* current = table[i];
1305        LVT_Hash* next;
1306        while (current != NULL) {
(gdb) 
1307          next = current->_next;  <----===== Why didn't this segfault?
1308          current->_next = NULL;  <----===== or here?
1309          delete(current);  <--------==============   HERE
1310          current = next;
1311        }
1312        table[i] = NULL;
1313      }
1314    }
1315
1316    LVT_Hash* LVT_lookup(LocalVariableTableElement *elem, int index, LVT_Hash** table) {

(gdb) p current
$17 = (LVT_Hash *) 0x6ed0
(gdb) p *current
Cannot access memory at address 0x6ed0
(gdb) p current->_next
Cannot access memory at address 0x6ed8

So based on the current value of `current` we should have segfaulted on line 1307 or 1308 but we didn't. That makes me think the value of `current` changed between line 1308 and 1309.

Looking at the disassembly for lines 1303 and 1304 and comparing them to register values.

(gdb) inf reg
rax            0x0      0
rbx            0x0      0
rcx            0xffffffffffffffff       -1
rdx            0x6      6
rsi            0x6f44   28484
rdi            0x6ed0   28368
rbp            0x42d93420       0x42d93420
rsp            0x42d93400       0x42d93400
r8             0xffffffff       4294967295
r9             0x101010101010101        72340172838076673
r10            0x8      8
r11            0x202    514
r12            0xf2     242
r13            0x25120140       621936960
r14            0x1      1
r15            0x2439d7a2       607770530
rip            0x2b5eef2c1bf1   0x2b5eef2c1bf1 <clear_hashtable(LVT_Hash**)+49>
eflags         0x202    [ IF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x63     99
gs             0x0      0
fctrl          0x37f    895
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
mxcsr          0x1fa0   [ PE IM DM ZM OM UM PM ]


1303      for (int i = 0; i < HASH_ROW_SIZE; i++) {
0x00002b5eef2c1c06 <_Z15clear_hashtablePP8LVT_Hash+70>: cmp    $0x100,%r12
0x00002b5eef2c1c0d <_Z15clear_hashtablePP8LVT_Hash+77>: jne    0x2b5eef2c1bd3 <_Z15clear_hashtablePP8LVT_Hash+19>

1304        LVT_Hash* current = table[i];
0x00002b5eef2c1bd3 <_Z15clear_hashtablePP8LVT_Hash+19>: mov    0x0(%r13,%r12,8),%rdi

cmp    $0x100,%r12                 <-- Compare %r12, i (242) to 256 (0x100)
mov    0x0(%r13,%r12,8),%rdi       <-- Move the value at %r13 (0x25120140) + (%r12 (242) * 8) into %rdi 

(gdb) p/x (0xf2*0x8)+0x25120140
$20 = 0x251208d0
(gdb) x 0x251208d0
0x251208d0:     0x21bf0220

We can confirm this another way.

(gdb) p *table@256
$19 = {0x0 <repeats 242 times>, 0x21bf0220, 0x0 <repeats 13 times>}  <--- Note the sole non-null value.

(gdb) p table[242]
$21 = (LVT_Hash *) 0x21bf0220

(gdb) p *table[242]
$22 = {<CHeapObj> = {<No data fields>}, _elem = 0x927604a6, _next = 0x0}

So the value in %rdi should have been 0x21bf0220 but is actually 0x6ed0. I'm not sure how this has happened as there does not appear to be any other thread in this code or near it.

We were working with this jar file (I'll attach it, the core and an sosreport).

jar:file:/opt/jboss-eap-5.1/jboss-as/common/lib/jboss-messaging.jar 


Version-Release number of selected component (if applicable):
java-1.6.0-openjdk-1.6.0.0-1.41.1.11.11.90.el5_9

How reproducible:
Unknown, seen once.
Comment 1 Brad Hubbard 2013-10-07 23:17:39 EDT
Created attachment 809057 [details]
Full backtrace of all threads
Comment 2 Brad Hubbard 2013-10-07 23:18:53 EDT
Created attachment 809058 [details]
jboss file being accessed at abort time
Comment 3 Brad Hubbard 2013-10-07 23:19:53 EDT
Created attachment 809059 [details]
sosreport
Comment 4 Brad Hubbard 2013-10-07 23:22:08 EDT
Core is, of course, too big to attach. Will leave it available here for some time.

http://people.redhat.com/bhubbard/core.28368.gz
Comment 6 RHEL Product and Program Management 2014-01-22 11:24:40 EST
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.
Comment 8 RHEL Product and Program Management 2014-01-28 12:16:38 EST
Development Management has reviewed and declined this request.
You may appeal this decision by reopening this request.

Note You need to log in before you can comment on or make changes to this bug.