Bug 734729

Description of problem:
There was detected qpidd broker crash during automation testing of bug 637751. In some cases qpidd broker generates "Segmentation fault" (core dumped) after receiving signal 2 (SIGINT). This behavior was seen on RHEL 6.1, but it was not seen on RHEL 5.7. All core dumps looks like this:

  #0  0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6
  #1  0xb53e1008 in ?? ()
  #2  0x005981a7 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/libstdc++.so.6
  #3  0x00921f28 in ObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:51

(full analysis is part of Additional info)

Version-Release number of selected component (if applicable):
RHEL 6.1 (i386, x86_64)
python-qpid-0.10-1.el6.noarch
python-qpid-qmf-0.10-10.el6.i686
qpid-cpp-client-0.10-6.el6.i686
qpid-cpp-client-devel-0.10-6.el6.i686
qpid-cpp-client-devel-docs-0.10-6.el6.noarch
qpid-cpp-client-rdma-0.10-6.el6.i686
qpid-cpp-client-ssl-0.10-6.el6.i686
qpid-cpp-debuginfo-0.10-6.el6.i686
qpid-cpp-server-0.10-6.el6.i686
qpid-cpp-server-devel-0.10-6.el6.i686
qpid-cpp-server-rdma-0.10-6.el6.i686
qpid-cpp-server-ssl-0.10-6.el6.i686
qpid-cpp-server-store-0.10-6.el6.i686
qpid-cpp-server-xml-0.10-6.el6.i686
qpid-java-client-0.10-6.el6.noarch
qpid-java-common-0.10-6.el6.noarch
qpid-java-example-0.10-6.el6.noarch
qpid-java-jca-0.10-6.el6.noarch
qpid-qmf-0.10-10.el6.i686
qpid-qmf-debuginfo-0.10-10.el6.i686
qpid-qmf-devel-0.10-10.el6.i686
qpid-tests-0.10-1.el6.noarch
qpid-tools-0.10-5.el6.noarch
rh-qpid-cpp-tests-0.10-6.el6.i686
ruby-qpid-0.7.946106-2.el6.i686
ruby-qpid-qmf-0.10-10.el6.i686


How reproducible:
10%

Steps to Reproduce:
1. Start standalone qpidd (--auth no --truncate yes --log-enable info+)
2. Create queue (qpid-config add queue qname_test --file-count=64 --file-size=1 --durable --cluster-durable)
3. List queue (qpid-config queues qname_test)
4. Stop qpidd by signal 2 (SIGINT)
  
Actual results:
qpidd broker code dumped (SIGSEGV).

Expected results:
qpidd broker stopped correctly.

Additional info:

[19:14:51] ==================== file-count=3, file-size=1 ==================
[19:14:51] mrg_qpidd_start: qpidd launched normal bg way (port:5672,log:qpidd.transcript.log,params: --auth no --truncate yes --log-enable info+ --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpid-config_check_params_bz637751/rhts_qpidd/20110830_191424/broker.UYHn,lockfile:qpidd.lock)
[19:14:53] mrg_qpidd_wait_on_settle: qpidd started-up (dur:1sec)
[19:14:53] .qpidd settled
[19:14:53] qpid-config add queue qname_test --file-count=3 --file-size=1 --durable --cluster-durable
[19:14:55] .ecode=0
[19:14:55] lf_run_grep file-count=8 file-size=1
[19:14:55] qpid-config queues qname_test
Queue Name                                             Attributes
===========================================================================
qname_test                                             --durable --cluster-durable --file-size=1 --file-count=3 
[19:14:57] .ecode=0 ok
[19:14:57] qpid-config queues qname_test |grep qname_test | grep file-count=8 | grep file-size=1
[19:14:59] .ERROR: ecode=1 test will fail
[19:14:59] Test may consume a lot of disk space. Verify size of journal dir:
[19:14:59] du -sh /var/lib/qpidd
30M	/var/lib/qpidd
[19:14:59] .ecode=0 ok
[19:14:59] Verify if any free space left on disk:
[19:14:59] df -hT /
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/vda1     ext4    8.3G  2.9G  5.0G  37% /
[19:14:59] .ecode=0 ok
[19:14:59] .qpidd status check
[19:14:59] mrg_qpidd_status: 1 instance[s] running (pids:12684 ,ports: 5672)
[19:14:59] ..qpidd running fine on port  5672 (1 instance[s])
[19:14:59] .qpidd stop
[19:14:59] mrg_qpidd_stop: following qpidd brokers are found to stop (pid[s]:12684 )
/root/tests/distribution/MRG/Messaging/qpid_common/qpid_test_functions_broker.sh: line 53: 12684 Segmentation fault      (core dumped) ${qpidd_wrapper} ${QPIDD_DIR}/${MRG_CPP_BROKER} ${datadir_sw} --port ${port} ${test_params} &>${transcript}
[19:14:59] mrg_kill_process_id:processes  12684 stopped by signal[s] 2
[19:14:59] .qpidd stopped ok
[19:14:59] ERROR:core test failed! file-count=3, file-size=1

[20:09:39] Core file: ./core.12684 generated by /usr/sbin/qpidd ----------------------1/8-
-rw-------. 1 root root 36704256 Aug 30 19:14 ./core.12684
./core.12684: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from '/usr/sbin/qpidd --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpi'
  GNU gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
  Copyright (C) 2010 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "i686-redhat-linux-gnu".
  For bug reporting instructions, please see:
  Missing separate debuginfo for 
  Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/9d/1fdcc265b5b59e5723578770e1c7ba9015e9a6
  [New Thread 12684]
  [Thread debugging using libthread_db enabled]
  Core was generated by `/usr/sbin/qpidd --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpi'.
  Program terminated with signal 11, Segmentation fault.
  #0  0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6
  Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.i686 boost-program-options-1.41.0-11.el6_1.2.i686 boost-system-1.41.0-11.el6_1.2.i686 cyrus-sasl-lib-2.1.23-8.el6.i686 db4-cxx-4.7.25-16.el6.i686 glibc-2.12-1.25.el6_1.3.i686 libaio-0.3.107-10.el6.i686 libgcc-4.4.5-6.el6.i686 libibverbs-1.1.4-2.el6.i686 librdmacm-1.0.10-2.el6.i686 libstdc++-4.4.5-6.el6.i686 libuuid-2.17.2-12.el6.i686 nspr-4.8.7-1.el6.i686 nss-3.12.9-9.el6.i686 nss-softokn-freebl-3.12.9-3.el6.i686 nss-util-3.12.9-1.el6.i686 xerces-c-3.0.1-20.el6.i686 xqilla-2.2.3-8.el6.i686 zlib-1.2.3-25.el6.i686
  (gdb) eax            0x0	0
  ecx            0x301a8f	3152527
  edx            0x86315f8	140711416
  ebx            0x5edff4	6217716
  esp            0xbf809998	0xbf809998
  ebp            0xbf8099d8	0xbf8099d8
  esi            0x8649000	140808192
  edi            0xb53f8a10	-1254127088
  eip            0xb09b56	0xb09b56 <__memcpy_ia32+70>
  eflags         0x10246	[ PF ZF IF RF ]
  cs             0x73	115
  ss             0x7b	123
  ds             0x7b	123
  es             0x7b	123
  fs             0x0	0
  gs             0x33	51
  (gdb) Using memory regions provided by the target.
  There are no memory regions defined.
  (gdb) 32   AT_SYSINFO           Special system info/entry points 0xe9e414
  33   AT_SYSINFO_EHDR      System-supplied DSO's ELF header 0xe9e000
  16   AT_HWCAP             Machine-dependent CPU capability hints 0x78bfbfd
  6    AT_PAGESZ            System page size               4096
  17   AT_CLKTCK            Frequency of times()           100
  3    AT_PHDR              Program headers for program    0x8048034
  4    AT_PHENT             Size of program header entry   32
  5    AT_PHNUM             Number of program headers      8
  7    AT_BASE              Base address of interpreter    0x49a000
  8    AT_FLAGS             Flags                          0x0
  9    AT_ENTRY             Entry point of program         0x80506d0
  11   AT_UID               Real user ID                   0
  12   AT_EUID              Effective user ID              0
  13   AT_GID               Real group ID                  0
  14   AT_EGID              Effective group ID             0
  23   AT_SECURE            Boolean, was exec setuid-like? 0
  25   AT_RANDOM            Address of 16 random bytes     0xbf80a30b
  31   AT_EXECFN            File name of executable        0xbf80cfec "/usr/sbin/qpidd"
  15   AT_PLATFORM          String identifying platform    0xbf80a31b "i686"
  0    AT_NULL              End of vector                  0x0
  (gdb) Stack level 0, frame at 0xbf80999c:
   eip = 0xb09b56 in __memcpy_ia32; saved eip 0xb53e1008
   called by frame at 0xbf8099e0
   Arglist at 0xbf809994, args: 
   Locals at 0xbf809994, Previous frame's sp is 0xbf80999c
   Saved registers:
    eip at 0xbf809998
  (gdb) From        To          Syms Read   Shared Object Library
  0x007add80  0x0094b798  Yes         /usr/lib/libqpidbroker.so.5.0.0
  0x001ee1b0  0x002ed218  Yes         /usr/lib/libqpidcommon.so.5.0.0
  0x003740f0  0x003820c8  Yes         /usr/lib/libqpidtypes.so.1.2.0
  0x00a31390  0x00a4e668  Yes (*)     /usr/lib/libboost_program_options.so.5
  0x0038fe50  0x003973f8  Yes (*)     /usr/lib/libboost_filesystem.so.5
  0x00506fc0  0x00508be8  Yes (*)     /lib/libuuid.so.1
  0x00f1ca60  0x00f1da88  Yes (*)     /lib/libdl.so.2
  0x00a89880  0x00a8d608  Yes (*)     /lib/librt.so.1
  0x0039e1b0  0x003ae9d8  Yes (*)     /usr/lib/libsasl2.so.2
  0x0054deb0  0x005c0898  Yes (*)     /usr/lib/libstdc++.so.6
  0x003b7470  0x003d1968  Yes (*)     /lib/libm.so.6
  0x003dffa0  0x003f83c8  Yes (*)     /lib/libgcc_s.so.1
  0x00aa7ac0  0x00bca754  Yes (*)     /lib/libc.so.6
  0x00e2c5e0  0x00e38518  Yes (*)     /lib/libpthread.so.0
  0x0049a830  0x004b20df  Yes (*)     /lib/ld-linux.so.2
  0x003fccc0  0x003fdab8  Yes (*)     /usr/lib/libboost_system.so.5
  0x00de0650  0x00deeed8  Yes (*)     /lib/libresolv.so.2
  0x00c788b0  0x00c7da18  Yes (*)     /lib/libcrypt.so.1
  0x004019d0  0x00433ae8  Yes (*)     /lib/libfreebl3.so
  0x00626a30  0x006bfdb8  Yes         /usr/lib/qpid/daemon/msgstore.so
  0x00f46830  0x01076d38  Yes (*)     /usr/lib/libdb_cxx-4.7.so
  0x00e633d0  0x00e63720  Yes (*)     /lib/libaio.so.1
  0x00a78d10  0x00a83688  Yes         /usr/lib/qpid/daemon/xml.so
  0x078964e0  0x07a19c48  Yes (*)     /usr/lib/libxerces-c-3.0.so
  0x0430ad60  0x044f7008  Yes (*)     /usr/lib/libxqilla.so.5
  0x00450120  0x0045e8a8  Yes (*)     /lib/libnsl.so.1
  0x00e6f430  0x00e708e8  Yes (*)     /usr/lib/gconv/UTF-16.so
  0x00473990  0x0047bd98  Yes         /usr/lib/qpid/daemon/ssl.so
  0x004cb250  0x004e0bf8  Yes         /usr/lib/libsslcommon.so.5.0.0
  0x01575190  0x0165f448  Yes (*)     /usr/lib/libnss3.so
  0x009c3030  0x009e8928  Yes (*)     /usr/lib/libssl3.so
  0x00ec5800  0x00ee9328  Yes (*)     /lib/libnspr4.so
  0x004eec90  0x004faba8  Yes (*)     /usr/lib/libnssutil3.so
  0x00eb3e50  0x00eb5818  Yes (*)     /lib/libplc4.so
  0x00481ab0  0x00482a98  Yes (*)     /lib/libplds4.so
  0x00d67620  0x00d72888  Yes (*)     /lib/libz.so.1
  0x006f0940  0x006f8558  Yes         /usr/lib/qpid/daemon/rdma.so
  0x009ff2d0  0x00a0ba38  Yes         /usr/lib/librdmawrap.so.5.0.0
  0x00486050  0x00488fe8  Yes (*)     /usr/lib/librdmacm.so.1
  0x0048d510  0x004942a8  Yes (*)     /usr/lib/libibverbs.so.1
  0x00a63c30  0x00a69208  Yes         /usr/lib/qpid/daemon/replicating_listener.so
  0x00701ee0  0x00707438  Yes         /usr/lib/qpid/daemon/replication_exchange.so
  0x00c2f910  0x00c4e038  Yes         /usr/lib/qpid/daemon/acl.so
  (*): Shared library is missing debugging information.
  (gdb) * 1 Thread 0xb7890730 (LWP 12684)  0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6
  Thread 1 (Thread 0xb7890730 (LWP 12684)):
  #0  0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6
  #1  0xb53e1008 in ?? ()
  #2  0x005981a7 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/libstdc++.so.6
  #3  0x00921f28 in ObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:51
  #4  getObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:199
  #5  qpid::management::ManagementAgent::RemoteAgent::~RemoteAgent (this=0x862fee0, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:113
  #6  0x009221d3 in qpid::management::ManagementAgent::RemoteAgent::~RemoteAgent (this=0x862fee0, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:115
  #7  0x0093ecb9 in checked_delete<qpid::management::ManagementAgent::RemoteAgent> (this=0x862ff20) at /usr/include/boost/checked_delete.hpp:34
  #8  boost::detail::sp_counted_impl_p<qpid::management::ManagementAgent::RemoteAgent>::dispose (this=0x862ff20) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
  #9  0x0082a910 in release (this=0x8631418, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
  #10 boost::detail::shared_count::~shared_count (this=0x8631418, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
  #11 0x00941199 in std::_Rb_tree<qpid::management::ObjectId, std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> >, std::_Select1st<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > >, std::less<qpid::management::ObjectId>, std::allocator<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > > >::_M_erase(std::_Rb_tree_node<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > >*) () from /usr/lib/libqpidbroker.so.5.0.0
  #12 0x009321f9 in ~_Rb_tree (this=0xb6e5c008, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/bits/stl_tree.h:614
  #13 ~map (this=0xb6e5c008, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/bits/stl_map.h:87
  #14 qpid::management::ManagementAgent::~ManagementAgent (this=0xb6e5c008, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:158
  #15 0x00932553 in qpid::management::ManagementAgent::~ManagementAgent (this=0xb6e5c008, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:158
  #16 0x00839bca in ~auto_ptr (this=0x8582230, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/backward/auto_ptr.h:168
  #17 qpid::broker::Broker::~Broker (this=0x8582230, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:405
  #18 0x0083a403 in qpid::broker::Broker::~Broker (this=0x8582230, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:405
  #19 0x0082a866 in qpid::RefCounted::released (this=0x8582248) at qpid/RefCounted.h:48
  #20 0x080547ec in release (this=0xbf80a05f, options=0x8579b20) at qpid/RefCounted.h:42
  #21 intrusive_ptr_release (this=0xbf80a05f, options=0x8579b20) at qpid/RefCounted.h:57
  #22 ~intrusive_ptr (this=0xbf80a05f, options=0x8579b20) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
  #23 QpiddBroker::execute (this=0xbf80a05f, options=0x8579b20) at posix/QpiddBroker.cpp:187
  #24 0x08050c81 in main (argc=-172655616, argv=0x2eff4ff) at qpidd.cpp:80
  (gdb) quit