Bug 590624 - qpidd broker crash in mrg::msgstore::JournalImpl::wr_aio_cb() -> release() -> free()
Summary: qpidd broker crash in mrg::msgstore::JournalImpl::wr_aio_cb() -> release() ->...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: qpid-cpp
Version: Development
Hardware: All
OS: Linux
urgent
high
Target Milestone: 1.3
: ---
Assignee: Kim van der Riet
QA Contact: Frantisek Reznicek
URL:
Whiteboard:
Keywords:
: 590636 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2010-05-10 11:02 UTC by Frantisek Reznicek
Modified: 2015-11-16 01:12 UTC (History)
2 users (show)

(edit)
Clone Of:
(edit)
Last Closed:


Attachments (Terms of Use)

Description Frantisek Reznicek 2010-05-10 11:02:28 UTC
Description of problem:

I have found qpidd crashing in mrg::msgstore::JournalImpl::wr_aio_cb() -> release() -> free() call path. The issue is likely close to bug 587505, but broker in this case crashes, while the bug 587505 causes broker to abort.


Observed on RHEL 5.5 i386:

The part of the backtrace

Thread 1 (Thread 7312):
#0  0x00000036a1071d80 in _int_free () from /lib64/libc.so.6
#1  0x00000036a107276b in free () from /lib64/libc.so.6
#2  0x00002ba62b503d70 in release (this=<value optimized out>, dtokl=...)
    at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42
#3  mrg::msgstore::JournalImpl::wr_aio_cb (this=<value optimized out>,
    dtokl=...) at JournalImpl.cpp:584
#4  0x00002ba62b5764a7 in mrg::journal::wmgr::get_events (this=0xf5a54b8,
    state=UNUSED) at jrnl/wmgr.cpp:755
#5  0x00002ba62b578d7f in mrg::journal::wmgr::write_flush (this=0xf5a54b8)
    at jrnl/wmgr.cpp:615
...


Version-Release number of selected component (if applicable):


How reproducible:
very hard (<2%)

Steps to Reproduce:
1. run qpid_test_qpidd-perftest_performance and wait for crash
  launch broker
  loop the perftest client to sweep the parameters
  keep running until broker crashes
  
Actual results:
qpidd broker crashes.

Expected results:
qpidd broker should not crash.

Additional info:

observed broker crash:
~~~~~~~~~~~~~~~~~~~~~~

[root@mrg-qe-02 qpid_test_qpidd-perftest_performance]# cat dump_core.7312
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/qpidd...Reading symbols from /usr/lib/debug/usr/sbin/qpidd.debug...
warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .dynbss not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/qpidd.debug
done.
done.
[New Thread 7326]
[New Thread 7325]
[New Thread 7324]
[New Thread 7323]
[New Thread 7322]
[New Thread 7321]
[New Thread 7320]
[New Thread 7319]
[New Thread 7318]
[New Thread 7317]

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .dynbss not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/qpidd.debug
Reading symbols from /usr/lib64/libqpidbroker.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidbroker.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidbroker.so.2
Reading symbols from /usr/lib64/libqpidcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidcommon.so.2
Reading symbols from /usr/lib64/libboost_program_options.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_program_options.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_program_options.so.2
Reading symbols from /usr/lib64/libboost_filesystem.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libboost_filesystem.so.1.33.1.debug...done.
done.
Loaded symbols for /usr/lib64/libboost_filesystem.so.2
Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/qpid/daemon/acl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/acl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/acl.so
Reading symbols from /usr/lib64/qpid/daemon/xml.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/xml.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/xml.so
Reading symbols from /usr/lib64/libxerces-c.so.28...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxerces-c.so.28
Reading symbols from /usr/lib64/libxqilla.so.3...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libxqilla.so.3
Reading symbols from /usr/lib64/qpid/daemon/cluster.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/cluster.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/cluster.so
Reading symbols from /usr/lib64/openais/libcpg.so.2...Reading symbols from /usr/lib/debug/usr/lib64/openais/libcpg.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/openais/libcpg.so.2
Reading symbols from /usr/lib64/libcman.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcman.so.2
Reading symbols from /usr/lib64/libqpidclient.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libqpidclient.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libqpidclient.so.2
Reading symbols from /usr/lib64/qpid/client/sslconnector.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/client/sslconnector.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/client/sslconnector.so
Reading symbols from /usr/lib64/libsslcommon.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libsslcommon.so.2.0.0.debug...done.
done.
Loaded symbols for /usr/lib64/libsslcommon.so.2
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnspr4.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /usr/lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplc4.so
Reading symbols from /usr/lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libplds4.so
Reading symbols from /usr/lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libz.so.1
Reading symbols from /usr/lib64/qpid/daemon/msgstore.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/msgstore.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/msgstore.so
Reading symbols from /usr/lib64/libdb_cxx-4.3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libdb_cxx-4.3.so
Reading symbols from /usr/lib64/libaio.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libaio.so.1
Reading symbols from /usr/lib64/qpid/daemon/ssl.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/ssl.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/ssl.so
Reading symbols from /usr/lib64/qpid/daemon/watchdog.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/watchdog.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/watchdog.so
Reading symbols from /usr/lib64/qpid/daemon/replication_exchange.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replication_exchange.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replication_exchange.so
Reading symbols from /usr/lib64/qpid/daemon/replicating_listener.so...Reading symbols from /usr/lib/debug/usr/lib64/qpid/daemon/replicating_listener.so.debug...done.
done.
Loaded symbols for /usr/lib64/qpid/daemon/replicating_listener.so

warning: section .gnu.liblist not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .dynbss not found in /usr/lib/debug/usr/sbin/qpidd.debug

warning: section .gnu.conflict not found in /usr/lib/debug/usr/sbin/qpidd.debug
Core was generated by `/usr/sbin/qpidd --data-dir /root/MRG/Messaging/qpid_test_qpidd-perftest_perform'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000036a1071d80 in _int_free () from /lib64/libc.so.6
(gdb) rax            0x1000100  16777472
rbx            0x2aaaac326630   46912521791024
rcx            0x2aaaac326690   46912521791120
rdx            0x1f00010b59000000       2233786563425206272
rsi            0x1      1
rdi            0x2aaaac000020   46912518488096
rbp            0x100    0x100
rsp            0x7ffff8727110   0x7ffff8727110
r8             0x2aaaac2df090   46912521498768
r9             0x1c90   7312
r10            0xf6b23f0        258679792
r11            0x36a1c08740     234641983296
r12            0x2aaaac326730   46912521791280
r13            0x36a1122418     234630554648
r14            0x430    1072
r15            0x2aaaac000020   46912518488096
rip            0x36a1071d80     0x36a1071d80 <_int_free+400>
eflags         0x10246  [ PF ZF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            0        (raw 0x00000000000000000000)
st3            0        (raw 0x00000000000000000000)
st4            0        (raw 0x00000000000000000000)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            0        (raw 0x00000000000000000000)
fctrl          0x0      0
fstat          0x0      0
ftag           0x0      0
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
(gdb) Using memory regions provided by the target.
There are no memory regions defined.
(gdb) From                To                  Syms Read   Shared Object Library
0x000000358809f1a0  0x00000035882230b8  Yes (*)     /usr/lib64/libqpidbroker.so.2
0x0000003c5630af10  0x0000003c5640bc38  Yes (*)     /usr/lib64/libqpidcommon.so.2
0x0000003674010aa0  0x000000367402dae8  Yes (*)     /usr/lib64/libboost_program_options.so.2
0x0000003674404810  0x000000367440cff8  Yes (*)     /usr/lib64/libboost_filesystem.so.2
0x00000036b5e01500  0x00000036b5e02918  Yes (*)     /lib64/libuuid.so.1
0x00000036a1800e10  0x00000036a1801a08  Yes (*)     /lib64/libdl.so.2
0x00000036a2402220  0x00000036a2405cc8  Yes (*)     /lib64/librt.so.1
0x00000036b66046e0  0x00000036b6613be8  Yes (*)     /usr/lib64/libsasl2.so.2
0x00000036b3a4f430  0x00000036b3ac3058  Yes (*)     /usr/lib64/libstdc++.so.6
0x00000036a1403e60  0x00000036a1443e38  Yes (*)     /lib64/libm.so.6
0x00000036af201e50  0x00000036af20b018  Yes (*)     /lib64/libgcc_s.so.1
0x00000036a101d780  0x00000036a1109ff8  Yes (*)     /lib64/libc.so.6
0x00000036a0c00a70  0x00000036a0c1671e  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00000036a1c051f0  0x00000036a1c10258  Yes (*)     /lib64/libpthread.so.0
0x00000036a54032a0  0x00000036a540e2d8  Yes (*)     /lib64/libresolv.so.2
0x00000036b1e009f0  0x00000036b1e06918  Yes (*)     /lib64/libcrypt.so.1
0x00002ba629f0abf0  0x00002ba629f29f98  Yes (*)     /usr/lib64/qpid/daemon/acl.so
0x00002ba62a13c7f0  0x00002ba62a146988  Yes (*)     /usr/lib64/qpid/daemon/xml.so
0x0000003673773070  0x00000036738f4758  Yes (*)     /usr/lib64/libxerces-c.so.28
0x00002ba62a4cf090  0x00002ba62a65ab28  Yes (*)     /usr/lib64/libxqilla.so.3
0x00002ba62a9d3da0  0x00002ba62aa3f528  Yes (*)     /usr/lib64/qpid/daemon/cluster.so
0x00002ba62ac7d3d0  0x00002ba62ac7f338  Yes (*)     /usr/lib64/openais/libcpg.so.2
0x00002ba62ae81110  0x00002ba62ae83b78  Yes (*)     /usr/lib64/libcman.so.2
0x0000003c5685f120  0x0000003c56922078  Yes (*)     /usr/lib64/libqpidclient.so.2
0x00002ba62b08e880  0x00002ba62b09b248  Yes (*)     /usr/lib64/qpid/client/sslconnector.so
0x00002ba62b2ac350  0x00002ba62b2c3058  Yes (*)     /usr/lib64/libsslcommon.so.2
0x00000030b8c183b0  0x00000030b8cf6f08  Yes (*)     /usr/lib64/libnss3.so
0x00000030b98085e0  0x00000030b982b638  Yes (*)     /usr/lib64/libssl3.so
0x00000030b800cf30  0x00000030b802b738  Yes (*)     /usr/lib64/libnspr4.so
0x00000030b9008340  0x00000030b9012c38  Yes (*)     /usr/lib64/libnssutil3.so
0x00000030b8401370  0x00000030b8402978  Yes (*)     /usr/lib64/libplc4.so
0x00000030b8800e10  0x00000030b8801c08  Yes (*)     /usr/lib64/libplds4.so
0x00000036a2001fd0  0x00000036a200cac8  Yes (*)     /usr/lib64/libz.so.1
0x00002ba62b500d10  0x00002ba62b5905d8  Yes (*)     /usr/lib64/qpid/daemon/msgstore.so
0x00002ba62b7fa5d0  0x00002ba62b8b3288  Yes (*)     /usr/lib64/libdb_cxx-4.3.so
0x00002ba62bade510  0x00002ba62bade6d1  Yes (*)     /usr/lib64/libaio.so.1
0x00002ba62bce8570  0x00002ba62bcf0698  Yes (*)     /usr/lib64/qpid/daemon/ssl.so
0x00002ba62befa5e0  0x00002ba62befdc18  Yes (*)     /usr/lib64/qpid/daemon/watchdog.so
0x00002ba62c105620  0x00002ba62c109a68  Yes (*)     /usr/lib64/qpid/daemon/replication_exchange.so
0x00002ba62c312b70  0x00002ba62c318708  Yes (*)     /usr/lib64/qpid/daemon/replicating_listener.so
(*): Shared library is missing debugging information.
(gdb)   11 Thread 7317  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  10 Thread 7318  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
  9 Thread 7319  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  8 Thread 7320  0x00000036a10d408a in epoll_ctl () from /lib64/libc.so.6
  7 Thread 7321  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  6 Thread 7322  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  5 Thread 7323  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  4 Thread 7324  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
  3 Thread 7325  0x00000036a10c5ff0 in __read_nocancel () from /lib64/libc.so.6
  2 Thread 7326  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
* 1 Thread 7312  0x00000036a1071d80 in _int_free () from /lib64/libc.so.6
(gdb)
Thread 11 (Thread 7317):
#0  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000003c564074d8 in qpid::sys::Timer::run (this=0xf4c4cf0)
    at ../include/qpid/sys/posix/Condition.h:69
#2  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0xf4c4d24) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 10 (Thread 7318):
#0  0x00000036a1c0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x0000003c564074d8 in qpid::sys::Timer::run (this=0xf4cee90)
    at ../include/qpid/sys/posix/Condition.h:69
#2  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0xf4ceec4) at qpid/sys/posix/Thread.cpp:35
#3  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#4  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 9 (Thread 7319):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 8 (Thread 7320):
#0  0x00000036a10d408a in epoll_ctl () from /lib64/libc.so.6
#1  0x0000003c563291ab in qpid::sys::PollerPrivate::resetMode (this=0xf4af5b0,
    eh=...) at qpid/sys/epoll/EpollPoller.cpp:352
#2  0x0000003c563294bb in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:512
#3  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#4  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#5  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#6  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 7 (Thread 7321):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 6 (Thread 7322):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 5 (Thread 7323):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 4 (Thread 7324):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 3 (Thread 7325):
#0  0x00000036a10c5ff0 in __read_nocancel () from /lib64/libc.so.6
#1  0x00000036a106a8eb in __libc_message () from /lib64/libc.so.6
#2  0x00000036a107230f in _int_free () from /lib64/libc.so.6
#3  0x00000036a107276b in free () from /lib64/libc.so.6
#4  0x00002ba62b5151d2 in deallocate (this=0xf4c6c10, queue=0x2aaaac042070,
    txn=0x468d2ff0, messageId=..., message=..., newId=240)
    at /usr/include/c++/4.1.2/ext/new_allocator.h:94
#5  _M_deallocate (this=0xf4c6c10, queue=0x2aaaac042070, txn=0x468d2ff0,
    messageId=..., message=..., newId=240)
    at /usr/include/c++/4.1.2/bits/stl_vector.h:133
#6  ~_Vector_base (this=0xf4c6c10, queue=0x2aaaac042070, txn=0x468d2ff0,
    messageId=..., message=..., newId=240)
    at /usr/include/c++/4.1.2/bits/stl_vector.h:119
#7  ~vector (this=0xf4c6c10, queue=0x2aaaac042070, txn=0x468d2ff0,
    messageId=..., message=..., newId=240)
    at /usr/include/c++/4.1.2/bits/stl_vector.h:272
#8  mrg::msgstore::MessageStoreImpl::store (this=0xf4c6c10,
    queue=0x2aaaac042070, txn=0x468d2ff0, messageId=..., message=...,
    newId=240) at MessageStoreImpl.cpp:1493
#9  0x00002ba62b52301b in mrg::msgstore::MessageStoreImpl::enqueue (
    this=0xf4c6c10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
#10 0x00000035881795cb in qpid::broker::MessageStoreModule::enqueue (
    this=<value optimized out>, ctxt=0x468d21c0, msg=..., queue=...)
    at qpid/broker/MessageStoreModule.cpp:124
#11 0x000000358818860c in qpid::broker::Queue::enqueue (this=0x2aaaac042070,
    ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>)
    at qpid/broker/Queue.cpp:753
#12 0x0000003588189dc6 in qpid::broker::Queue::deliver (this=0x2aaaac042070,
    msg=...) at qpid/broker/Queue.cpp:164
#13 0x000000358812c7e3 in qpid::broker::DeliverableMessage::deliverTo (
    this=0x468d4050, queue=...) at qpid/broker/DeliverableMessage.cpp:31
#14 0x0000003588144065 in qpid::broker::Exchange::doRoute (this=0xf4d0848,
    msg=..., b=...) at qpid/broker/Exchange.cpp:91
#15 0x000000358814dde7 in qpid::broker::FanOutExchange::route (this=0xf4d0848,
    msg=...) at qpid/broker/FanOutExchange.cpp:113
#16 0x00000035881babf3 in qpid::broker::SemanticState::route (
    this=<value optimized out>, msg=..., strategy=...)
    at qpid/broker/SemanticState.cpp:461
#17 0x00000035881bbb4d in qpid::broker::SemanticState::handle (this=0xf51dfc8,
    msg=...) at qpid/broker/SemanticState.cpp:415
#18 0x00000035881e090e in qpid::broker::SessionState::handleContent (
    this=0xf51ddf0, frame=..., id=<value optimized out>)
    at qpid/broker/SessionState.cpp:249
#19 0x00000035881e0eb0 in qpid::broker::SessionState::handleIn (
    this=0xf51ddf0, frame=...) at qpid/broker/SessionState.cpp:327
#20 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0xf51d890, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#21 0x000000358811dee2 in operator() (this=0xf51c280, frame=...)
    at qpid/framing/Handler.h:42
#22 qpid::broker::Connection::received (this=0xf51c280, frame=...)
    at qpid/broker/Connection.cpp:143
#23 0x00000035880fdd84 in qpid::amqp_0_10::Connection::decode (this=0xf51bd50,
    buffer=<value optimized out>, size=<value optimized out>)
    at qpid/amqp_0_10/Connection.cpp:58
#24 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff (
    this=0x2aaaac000c50, buff=0x2aaaac001390)
    at qpid/sys/AsynchIOHandler.cpp:135
#25 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
    this=0x400, a0=..., a1=0x400)
    at /usr/include/boost/function/function_template.hpp:576
#26 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable (
    this=0x2aaaac000e90, h=...) at qpid/sys/posix/AsynchIO.cpp:418
#27 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x400, a0=...)
    at /usr/include/boost/function/function_template.hpp:576
#28 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent (
    this=0x2aaaac000e98, type=<value optimized out>)
    at qpid/sys/DispatchHandle.cpp:278
#29 0x0000003c56329fff in process (this=0xf4a8eb0) at qpid/sys/Poller.h:123
#30 qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:483
#31 0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (
    p=0x9e) at qpid/sys/posix/Thread.cpp:35
#32 0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#33 0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 2 (Thread 7326):
#0  0x00000036a10d4108 in epoll_wait () from /lib64/libc.so.6
#1  0x0000003c563295af in qpid::sys::Poller::wait (this=0xf4a8eb0,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:524
#2  0x0000003c56329fd2 in qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:479
#3  0x0000003c563201ca in qpid::sys::(anonymous namespace)::runRunnable (p=0x8)
    at qpid/sys/posix/Thread.cpp:35
#4  0x00000036a1c0673d in start_thread () from /lib64/libpthread.so.0
#5  0x00000036a10d3d1d in clone () from /lib64/libc.so.6

Thread 1 (Thread 7312):
#0  0x00000036a1071d80 in _int_free () from /lib64/libc.so.6
#1  0x00000036a107276b in free () from /lib64/libc.so.6
#2  0x00002ba62b503d70 in release (this=<value optimized out>, dtokl=...)
    at /usr/src/debug/qpid-cpp-mrg-0.7.935473/cpp/src/qpid/RefCounted.h:42
#3  mrg::msgstore::JournalImpl::wr_aio_cb (this=<value optimized out>,
    dtokl=...) at JournalImpl.cpp:584
#4  0x00002ba62b5764a7 in mrg::journal::wmgr::get_events (this=0xf5a54b8,
    state=UNUSED) at jrnl/wmgr.cpp:755
#5  0x00002ba62b578d7f in mrg::journal::wmgr::write_flush (this=0xf5a54b8)
    at jrnl/wmgr.cpp:615
#6  0x00002ba62b57914d in mrg::journal::wmgr::flush_check (
    this=0x2aaaac000020, res=@0x7ffff8727bdc, cont=@0x7ffff8727be2,
    done=@0x7ffff8727be1) at jrnl/wmgr.cpp:530
#7  0x00002ba62b57a921 in mrg::journal::wmgr::enqueue (this=0xf5a54b8,
    data_buff=0x2aaaac2c0940, tot_data_len=1117,
    this_data_len=<value optimized out>, dtokp=0x2aaaac2c0db0, xid_ptr=0x0,
    xid_len=0, transient=false, external=false) at jrnl/wmgr.cpp:208
#8  0x00002ba62b5553be in mrg::journal::jcntl::enqueue_data_record (
    this=0xf5a5158, data_buff=0x2aaaac2c0940, tot_data_len=1117,
    this_data_len=1117, dtokp=0x2aaaac2c0db0, transient=<value optimized out>)
    at jrnl/jcntl.cpp:206
#9  0x00002ba62b50a283 in mrg::msgstore::JournalImpl::enqueue_data_record (
    this=0xf5a5150, data_buff=0x1, tot_data_len=2233786563425206272,
    this_data_len=46912521791120, dtokp=0x2aaaac2df090, transient=144)
    at JournalImpl.cpp:355
#10 0x00002ba62b5151a7 in mrg::msgstore::MessageStoreImpl::store (
    this=0xf4c6c10, queue=0xf5a45f0, txn=0x7ffff8728310, messageId=...,
    message=..., newId=<value optimized out>) at MessageStoreImpl.cpp:1474
#11 0x00002ba62b52301b in mrg::msgstore::MessageStoreImpl::enqueue (
    this=0xf4c6c10, ctxt=0x0, msg=..., queue=...) at MessageStoreImpl.cpp:1434
#12 0x00000035881795cb in qpid::broker::MessageStoreModule::enqueue (
    this=<value optimized out>, ctxt=0x1, msg=..., queue=...)
    at qpid/broker/MessageStoreModule.cpp:124
#13 0x000000358818860c in qpid::broker::Queue::enqueue (this=0xf5a45f0,
    ctxt=0x0, msg=..., suppressPolicyCheck=<value optimized out>)
    at qpid/broker/Queue.cpp:753
#14 0x0000003588189dc6 in qpid::broker::Queue::deliver (this=0xf5a45f0,
    msg=...) at qpid/broker/Queue.cpp:164
#15 0x000000358812c7e3 in qpid::broker::DeliverableMessage::deliverTo (
    this=0x7ffff8729370, queue=...) at qpid/broker/DeliverableMessage.cpp:31
#16 0x0000003588144065 in qpid::broker::Exchange::doRoute (this=0xf4d0848,
    msg=..., b=...) at qpid/broker/Exchange.cpp:91
#17 0x000000358814dde7 in qpid::broker::FanOutExchange::route (this=0xf4d0848,
    msg=...) at qpid/broker/FanOutExchange.cpp:113
#18 0x00000035881babf3 in qpid::broker::SemanticState::route (
    this=<value optimized out>, msg=..., strategy=...)
    at qpid/broker/SemanticState.cpp:461
#19 0x00000035881bbb4d in qpid::broker::SemanticState::handle (this=0xf51b928,
    msg=...) at qpid/broker/SemanticState.cpp:415
#20 0x00000035881e090e in qpid::broker::SessionState::handleContent (
    this=0xf51b750, frame=..., id=<value optimized out>)
    at qpid/broker/SessionState.cpp:249
#21 0x00000035881e0eb0 in qpid::broker::SessionState::handleIn (
    this=0xf51b750, frame=...) at qpid/broker/SessionState.cpp:327
#22 0x0000003c563b9fa9 in qpid::amqp_0_10::SessionHandler::handleIn (
    this=0xf515ae0, f=...) at qpid/amqp_0_10/SessionHandler.cpp:93
#23 0x000000358811dee2 in operator() (this=0xf516e70, frame=...)
    at qpid/framing/Handler.h:42
#24 qpid::broker::Connection::received (this=0xf516e70, frame=...)
    at qpid/broker/Connection.cpp:143
#25 0x00000035880fdd84 in qpid::amqp_0_10::Connection::decode (this=0xf5162e0,
    buffer=<value optimized out>, size=<value optimized out>)
    at qpid/amqp_0_10/Connection.cpp:58
#26 0x0000003c563fde11 in qpid::sys::AsynchIOHandler::readbuff (
    this=0xf4d10a0, buff=0xf4c6530) at qpid/sys/AsynchIOHandler.cpp:135
#27 0x0000003c5631de4f in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*, std::allocator<boost::function_base> >::operator() (
    this=0x1000100, a0=..., a1=0x1f00010b59000000)
    at /usr/include/boost/function/function_template.hpp:576
#28 0x0000003c5631c3b3 in qpid::sys::posix::AsynchIO::readable (
    this=0xf4d4490, h=...) at qpid/sys/posix/AsynchIO.cpp:418
#29 0x0000003c56404e97 in boost::function1<void, qpid::sys::DispatchHandle&, std::allocator<boost::function_base> >::operator() (this=0x1000100, a0=...)
    at /usr/include/boost/function/function_template.hpp:576
#30 0x0000003c563ffd2f in qpid::sys::DispatchHandle::processEvent (
    this=0xf4d4498, type=<value optimized out>)
    at qpid/sys/DispatchHandle.cpp:278
#31 0x0000003c56329fff in process (this=0xf4a8eb0) at qpid/sys/Poller.h:123
#32 qpid::sys::Poller::run (this=0xf4a8eb0)
    at qpid/sys/epoll/EpollPoller.cpp:483
#33 0x000000358810a2f2 in qpid::broker::Broker::run (
    this=<value optimized out>) at qpid/broker/Broker.cpp:334
#34 0x0000000000406ae6 in QpiddBroker::execute (this=<value optimized out>,
    options=0xf4a8160) at posix/QpiddBroker.cpp:176
#35 0x00000000004055af in main (argc=23, argv=0x7ffff872bc08) at qpidd.cpp:80
(gdb) quit

qpidd log says:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2010-05-08 11:12:03 info Loaded Module: /usr/lib64/qpid/daemon/acl.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/xml.so
2010-05-08 11:12:04 info SSL connector not enabled, you must set QPID_SSL_CERT_DB to enable it.
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/client/sslconnector.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/cluster.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/msgstore.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/ssl.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/watchdog.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/replication_exchange.so
2010-05-08 11:12:04 info Loaded Module: /usr/lib64/qpid/daemon/replicating_listener.so
2010-05-08 11:12:04 info No message store configured, persistence is disabled.
2010-05-08 11:12:04 info Management enabled
2010-05-08 11:12:04 info ManagementAgent generated broker ID: 0301e576-6151-4fb6-bcbd-b3d0fb41fa05
2010-05-08 11:12:04 info Registered xml exchange
2010-05-08 11:12:04 notice Store directory /root/MRG/Messaging/qpid_test_qpidd-perftest_performance/rhts_qpidd
/20100507_101011/b.5504/rhm was truncated.
2010-05-08 11:12:04 notice Journal "TplStore": Created
2010-05-08 11:12:04 notice Store module initialized; store-dir=/root/MRG/Messaging/qpid_test_qpidd-perftest_pe
rformance/rhts_qpidd/20100507_101011/b.5504
2010-05-08 11:12:04 info > Default files per journal: 64
2010-05-08 11:12:04 info > Default journal file size: 64 (wpgs)
2010-05-08 11:12:04 info > Default write cache page size: 64 (Kib)
2010-05-08 11:12:04 info > Default number of write cache pages: 16
2010-05-08 11:12:04 info > TPL files per journal: 64
2010-05-08 11:12:04 info > TPL journal file size: 64 (wpgs)
2010-05-08 11:12:04 info > TPL write cache page size: 16 (Kib)
2010-05-08 11:12:04 info > TPL number of write cache pages: 32
2010-05-08 11:12:04 info Registered replication exchange
2010-05-08 11:12:04 info Most recent persistence id found: 0x0
2010-05-08 11:12:04 notice SASL disabled: No Authentication Performed
2010-05-08 11:12:04 notice Listening on TCP port 41959
2010-05-08 11:12:04 info Policy file not specified. ACL Disabled, no ACL checking being done!
2010-05-08 11:12:04 info SSL plugin not enabled, you must set --ssl-cert-db to enable it.
2010-05-08 11:12:04 notice Broker running
2010-05-08 11:13:13 info Queue "perftest_pub_start": Policy created: type=reject; maxCount=0; maxSize=10485760
0
2010-05-08 11:13:13 info Queue "perftest_pub_done": Policy created: type=reject; maxCount=0; maxSize=104857600
2010-05-08 11:13:13 info Queue "perftest_sub_ready": Policy created: type=reject; maxCount=0; maxSize=10485760
0
2010-05-08 11:13:13 info Queue "perftest_sub_done": Policy created: type=reject; maxCount=0; maxSize=104857600
2010-05-08 11:13:13 notice Journal "anonymous.b77f0f45-93d1-4300-8431-f38a864a199b": Created
2010-05-08 11:13:13 notice Journal "anonymous.bc93cc26-ffa5-46e2-988e-4eb074ac815f": Created
2010-05-08 11:13:21 info Queue "anonymous.bc93cc26-ffa5-46e2-988e-4eb074ac815f": Policy created: type=reject;
maxCount=0; maxSize=104857600
2010-05-08 11:13:21 info Queue "anonymous.b77f0f45-93d1-4300-8431-f38a864a199b": Policy created: type=reject;
maxCount=0; maxSize=104857600


run 1 - perftest --iterations 1 --summary --unique-data yes --log-enable info+ -p 41959 --mode fanout --qt 1 -
-durable yes --npubs 2 --nsubs 2 --count 200000 --size 1024 --tx 0 --async-commit no log:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:13 info Connecting to tcp:localhost:41959
2010-05-08 11:13:47 warning Connection closed
2010-05-08 11:13:47 warning Connection closed
2010-05-08 11:13:47 warning Connection closed
2010-05-08 11:13:47 warning Connection closed
2010-05-08 11:13:47 warning Connection closed
PublishThread exception: Connection closed
SubscribeThread exception: Connection closedPublishThread exception: Connection closed

SubscribeThread exception: Connection closedController exception:
Connection closed

Comment 1 Frantisek Reznicek 2010-05-10 11:27:54 UTC
Package set used for above experiment is: qpid-cpp-*0.7.935473

Comment 2 Gordon Sim 2010-05-14 16:20:24 UTC
*** Bug 590636 has been marked as a duplicate of this bug. ***

Comment 3 Kim van der Riet 2010-05-17 15:28:09 UTC
There is a strong possibility that this is fixed by r944329 which reverts r736810 - and which has fixed several other similar bugs.

I have run a 1-hour soak test - repeating the perftest above in an endless loop; I have seen no occurrence. Setting to MODIFIED; if this should show up again, reopen the bug.

Comment 4 Frantisek Reznicek 2010-06-14 08:32:19 UTC
The issue has been fixed (no aborts / crashes detected), tested in four extended week runs on RHEL 4.8 / 5.5 i386 / x86_64 on packages:
python-qpid-0.7.946106-1.el5
python-saslwrapper-0.1.934605-2.el5
qpid-cpp-client-0.7.946106-2.el5
qpid-cpp-client-devel-0.7.946106-2.el5
qpid-cpp-client-devel-docs-0.7.946106-2.el5
qpid-cpp-client-ssl-0.7.946106-2.el5
qpid-cpp-mrg-debuginfo-0.7.946106-2.el5
qpid-cpp-server-0.7.946106-2.el5
qpid-cpp-server-cluster-0.7.946106-2.el5
qpid-cpp-server-devel-0.7.946106-2.el5
qpid-cpp-server-ssl-0.7.946106-2.el5
qpid-cpp-server-store-0.7.946106-2.el5
qpid-cpp-server-xml-0.7.946106-2.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tests-0.7.946106-1.el5
qpid-tools-0.7.946106-4.el5
ruby-qpid-0.7.946106-2.el5
ruby-saslwrapper-0.1.934605-2.el5
saslwrapper-0.1.934605-2.el5
saslwrapper-devel-0.1.934605-2.el5

-> VERIFIED


Note You need to log in before you can comment on or make changes to this bug.