Description of problem: There was detected qpidd broker crash during automation testing of bug 637751. In some cases qpidd broker generates "Segmentation fault" (core dumped) after receiving signal 2 (SIGINT). This behavior was seen on RHEL 6.1, but it was not seen on RHEL 5.7. All core dumps looks like this: #0 0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6 #1 0xb53e1008 in ?? () #2 0x005981a7 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/libstdc++.so.6 #3 0x00921f28 in ObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:51 (full analysis is part of Additional info) Version-Release number of selected component (if applicable): RHEL 6.1 (i386, x86_64) python-qpid-0.10-1.el6.noarch python-qpid-qmf-0.10-10.el6.i686 qpid-cpp-client-0.10-6.el6.i686 qpid-cpp-client-devel-0.10-6.el6.i686 qpid-cpp-client-devel-docs-0.10-6.el6.noarch qpid-cpp-client-rdma-0.10-6.el6.i686 qpid-cpp-client-ssl-0.10-6.el6.i686 qpid-cpp-debuginfo-0.10-6.el6.i686 qpid-cpp-server-0.10-6.el6.i686 qpid-cpp-server-devel-0.10-6.el6.i686 qpid-cpp-server-rdma-0.10-6.el6.i686 qpid-cpp-server-ssl-0.10-6.el6.i686 qpid-cpp-server-store-0.10-6.el6.i686 qpid-cpp-server-xml-0.10-6.el6.i686 qpid-java-client-0.10-6.el6.noarch qpid-java-common-0.10-6.el6.noarch qpid-java-example-0.10-6.el6.noarch qpid-java-jca-0.10-6.el6.noarch qpid-qmf-0.10-10.el6.i686 qpid-qmf-debuginfo-0.10-10.el6.i686 qpid-qmf-devel-0.10-10.el6.i686 qpid-tests-0.10-1.el6.noarch qpid-tools-0.10-5.el6.noarch rh-qpid-cpp-tests-0.10-6.el6.i686 ruby-qpid-0.7.946106-2.el6.i686 ruby-qpid-qmf-0.10-10.el6.i686 How reproducible: 10% Steps to Reproduce: 1. Start standalone qpidd (--auth no --truncate yes --log-enable info+) 2. Create queue (qpid-config add queue qname_test --file-count=64 --file-size=1 --durable --cluster-durable) 3. List queue (qpid-config queues qname_test) 4. Stop qpidd by signal 2 (SIGINT) Actual results: qpidd broker code dumped (SIGSEGV). Expected results: qpidd broker stopped correctly. Additional info: [19:14:51] ==================== file-count=3, file-size=1 ================== [19:14:51] mrg_qpidd_start: qpidd launched normal bg way (port:5672,log:qpidd.transcript.log,params: --auth no --truncate yes --log-enable info+ --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpid-config_check_params_bz637751/rhts_qpidd/20110830_191424/broker.UYHn,lockfile:qpidd.lock) [19:14:53] mrg_qpidd_wait_on_settle: qpidd started-up (dur:1sec) [19:14:53] .qpidd settled [19:14:53] qpid-config add queue qname_test --file-count=3 --file-size=1 --durable --cluster-durable [19:14:55] .ecode=0 [19:14:55] lf_run_grep file-count=8 file-size=1 [19:14:55] qpid-config queues qname_test Queue Name Attributes =========================================================================== qname_test --durable --cluster-durable --file-size=1 --file-count=3 [19:14:57] .ecode=0 ok [19:14:57] qpid-config queues qname_test |grep qname_test | grep file-count=8 | grep file-size=1 [19:14:59] .ERROR: ecode=1 test will fail [19:14:59] Test may consume a lot of disk space. Verify size of journal dir: [19:14:59] du -sh /var/lib/qpidd 30M /var/lib/qpidd [19:14:59] .ecode=0 ok [19:14:59] Verify if any free space left on disk: [19:14:59] df -hT / Filesystem Type Size Used Avail Use% Mounted on /dev/vda1 ext4 8.3G 2.9G 5.0G 37% / [19:14:59] .ecode=0 ok [19:14:59] .qpidd status check [19:14:59] mrg_qpidd_status: 1 instance[s] running (pids:12684 ,ports: 5672) [19:14:59] ..qpidd running fine on port 5672 (1 instance[s]) [19:14:59] .qpidd stop [19:14:59] mrg_qpidd_stop: following qpidd brokers are found to stop (pid[s]:12684 ) /root/tests/distribution/MRG/Messaging/qpid_common/qpid_test_functions_broker.sh: line 53: 12684 Segmentation fault (core dumped) ${qpidd_wrapper} ${QPIDD_DIR}/${MRG_CPP_BROKER} ${datadir_sw} --port ${port} ${test_params} &>${transcript} [19:14:59] mrg_kill_process_id:processes 12684 stopped by signal[s] 2 [19:14:59] .qpidd stopped ok [19:14:59] ERROR:core test failed! file-count=3, file-size=1 [20:09:39] Core file: ./core.12684 generated by /usr/sbin/qpidd ----------------------1/8- -rw-------. 1 root root 36704256 Aug 30 19:14 ./core.12684 ./core.12684: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from '/usr/sbin/qpidd --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpi' GNU gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: Missing separate debuginfo for Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/9d/1fdcc265b5b59e5723578770e1c7ba9015e9a6 [New Thread 12684] [Thread debugging using libthread_db enabled] Core was generated by `/usr/sbin/qpidd --data-dir /root/tests/distribution/MRG/Messaging/qpid_test_qpi'. Program terminated with signal 11, Segmentation fault. #0 0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6 Missing separate debuginfos, use: debuginfo-install boost-filesystem-1.41.0-11.el6_1.2.i686 boost-program-options-1.41.0-11.el6_1.2.i686 boost-system-1.41.0-11.el6_1.2.i686 cyrus-sasl-lib-2.1.23-8.el6.i686 db4-cxx-4.7.25-16.el6.i686 glibc-2.12-1.25.el6_1.3.i686 libaio-0.3.107-10.el6.i686 libgcc-4.4.5-6.el6.i686 libibverbs-1.1.4-2.el6.i686 librdmacm-1.0.10-2.el6.i686 libstdc++-4.4.5-6.el6.i686 libuuid-2.17.2-12.el6.i686 nspr-4.8.7-1.el6.i686 nss-3.12.9-9.el6.i686 nss-softokn-freebl-3.12.9-3.el6.i686 nss-util-3.12.9-1.el6.i686 xerces-c-3.0.1-20.el6.i686 xqilla-2.2.3-8.el6.i686 zlib-1.2.3-25.el6.i686 (gdb) eax 0x0 0 ecx 0x301a8f 3152527 edx 0x86315f8 140711416 ebx 0x5edff4 6217716 esp 0xbf809998 0xbf809998 ebp 0xbf8099d8 0xbf8099d8 esi 0x8649000 140808192 edi 0xb53f8a10 -1254127088 eip 0xb09b56 0xb09b56 <__memcpy_ia32+70> eflags 0x10246 [ PF ZF IF RF ] cs 0x73 115 ss 0x7b 123 ds 0x7b 123 es 0x7b 123 fs 0x0 0 gs 0x33 51 (gdb) Using memory regions provided by the target. There are no memory regions defined. (gdb) 32 AT_SYSINFO Special system info/entry points 0xe9e414 33 AT_SYSINFO_EHDR System-supplied DSO's ELF header 0xe9e000 16 AT_HWCAP Machine-dependent CPU capability hints 0x78bfbfd 6 AT_PAGESZ System page size 4096 17 AT_CLKTCK Frequency of times() 100 3 AT_PHDR Program headers for program 0x8048034 4 AT_PHENT Size of program header entry 32 5 AT_PHNUM Number of program headers 8 7 AT_BASE Base address of interpreter 0x49a000 8 AT_FLAGS Flags 0x0 9 AT_ENTRY Entry point of program 0x80506d0 11 AT_UID Real user ID 0 12 AT_EUID Effective user ID 0 13 AT_GID Real group ID 0 14 AT_EGID Effective group ID 0 23 AT_SECURE Boolean, was exec setuid-like? 0 25 AT_RANDOM Address of 16 random bytes 0xbf80a30b 31 AT_EXECFN File name of executable 0xbf80cfec "/usr/sbin/qpidd" 15 AT_PLATFORM String identifying platform 0xbf80a31b "i686" 0 AT_NULL End of vector 0x0 (gdb) Stack level 0, frame at 0xbf80999c: eip = 0xb09b56 in __memcpy_ia32; saved eip 0xb53e1008 called by frame at 0xbf8099e0 Arglist at 0xbf809994, args: Locals at 0xbf809994, Previous frame's sp is 0xbf80999c Saved registers: eip at 0xbf809998 (gdb) From To Syms Read Shared Object Library 0x007add80 0x0094b798 Yes /usr/lib/libqpidbroker.so.5.0.0 0x001ee1b0 0x002ed218 Yes /usr/lib/libqpidcommon.so.5.0.0 0x003740f0 0x003820c8 Yes /usr/lib/libqpidtypes.so.1.2.0 0x00a31390 0x00a4e668 Yes (*) /usr/lib/libboost_program_options.so.5 0x0038fe50 0x003973f8 Yes (*) /usr/lib/libboost_filesystem.so.5 0x00506fc0 0x00508be8 Yes (*) /lib/libuuid.so.1 0x00f1ca60 0x00f1da88 Yes (*) /lib/libdl.so.2 0x00a89880 0x00a8d608 Yes (*) /lib/librt.so.1 0x0039e1b0 0x003ae9d8 Yes (*) /usr/lib/libsasl2.so.2 0x0054deb0 0x005c0898 Yes (*) /usr/lib/libstdc++.so.6 0x003b7470 0x003d1968 Yes (*) /lib/libm.so.6 0x003dffa0 0x003f83c8 Yes (*) /lib/libgcc_s.so.1 0x00aa7ac0 0x00bca754 Yes (*) /lib/libc.so.6 0x00e2c5e0 0x00e38518 Yes (*) /lib/libpthread.so.0 0x0049a830 0x004b20df Yes (*) /lib/ld-linux.so.2 0x003fccc0 0x003fdab8 Yes (*) /usr/lib/libboost_system.so.5 0x00de0650 0x00deeed8 Yes (*) /lib/libresolv.so.2 0x00c788b0 0x00c7da18 Yes (*) /lib/libcrypt.so.1 0x004019d0 0x00433ae8 Yes (*) /lib/libfreebl3.so 0x00626a30 0x006bfdb8 Yes /usr/lib/qpid/daemon/msgstore.so 0x00f46830 0x01076d38 Yes (*) /usr/lib/libdb_cxx-4.7.so 0x00e633d0 0x00e63720 Yes (*) /lib/libaio.so.1 0x00a78d10 0x00a83688 Yes /usr/lib/qpid/daemon/xml.so 0x078964e0 0x07a19c48 Yes (*) /usr/lib/libxerces-c-3.0.so 0x0430ad60 0x044f7008 Yes (*) /usr/lib/libxqilla.so.5 0x00450120 0x0045e8a8 Yes (*) /lib/libnsl.so.1 0x00e6f430 0x00e708e8 Yes (*) /usr/lib/gconv/UTF-16.so 0x00473990 0x0047bd98 Yes /usr/lib/qpid/daemon/ssl.so 0x004cb250 0x004e0bf8 Yes /usr/lib/libsslcommon.so.5.0.0 0x01575190 0x0165f448 Yes (*) /usr/lib/libnss3.so 0x009c3030 0x009e8928 Yes (*) /usr/lib/libssl3.so 0x00ec5800 0x00ee9328 Yes (*) /lib/libnspr4.so 0x004eec90 0x004faba8 Yes (*) /usr/lib/libnssutil3.so 0x00eb3e50 0x00eb5818 Yes (*) /lib/libplc4.so 0x00481ab0 0x00482a98 Yes (*) /lib/libplds4.so 0x00d67620 0x00d72888 Yes (*) /lib/libz.so.1 0x006f0940 0x006f8558 Yes /usr/lib/qpid/daemon/rdma.so 0x009ff2d0 0x00a0ba38 Yes /usr/lib/librdmawrap.so.5.0.0 0x00486050 0x00488fe8 Yes (*) /usr/lib/librdmacm.so.1 0x0048d510 0x004942a8 Yes (*) /usr/lib/libibverbs.so.1 0x00a63c30 0x00a69208 Yes /usr/lib/qpid/daemon/replicating_listener.so 0x00701ee0 0x00707438 Yes /usr/lib/qpid/daemon/replication_exchange.so 0x00c2f910 0x00c4e038 Yes /usr/lib/qpid/daemon/acl.so (*): Shared library is missing debugging information. (gdb) * 1 Thread 0xb7890730 (LWP 12684) 0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6 Thread 1 (Thread 0xb7890730 (LWP 12684)): #0 0x00b09b56 in __memcpy_ia32 () from /lib/libc.so.6 #1 0xb53e1008 in ?? () #2 0x005981a7 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/libstdc++.so.6 #3 0x00921f28 in ObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:51 #4 getObjectId (this=0x862fee0, __in_chrg=<value optimized out>) at ../include/qpid/management/ManagementObject.h:199 #5 qpid::management::ManagementAgent::RemoteAgent::~RemoteAgent (this=0x862fee0, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:113 #6 0x009221d3 in qpid::management::ManagementAgent::RemoteAgent::~RemoteAgent (this=0x862fee0, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:115 #7 0x0093ecb9 in checked_delete<qpid::management::ManagementAgent::RemoteAgent> (this=0x862ff20) at /usr/include/boost/checked_delete.hpp:34 #8 boost::detail::sp_counted_impl_p<qpid::management::ManagementAgent::RemoteAgent>::dispose (this=0x862ff20) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78 #9 0x0082a910 in release (this=0x8631418, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145 #10 boost::detail::shared_count::~shared_count (this=0x8631418, __in_chrg=<value optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217 #11 0x00941199 in std::_Rb_tree<qpid::management::ObjectId, std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> >, std::_Select1st<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > >, std::less<qpid::management::ObjectId>, std::allocator<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > > >::_M_erase(std::_Rb_tree_node<std::pair<qpid::management::ObjectId const, boost::shared_ptr<qpid::management::ManagementAgent::RemoteAgent> > >*) () from /usr/lib/libqpidbroker.so.5.0.0 #12 0x009321f9 in ~_Rb_tree (this=0xb6e5c008, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/bits/stl_tree.h:614 #13 ~map (this=0xb6e5c008, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/bits/stl_map.h:87 #14 qpid::management::ManagementAgent::~ManagementAgent (this=0xb6e5c008, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:158 #15 0x00932553 in qpid::management::ManagementAgent::~ManagementAgent (this=0xb6e5c008, __in_chrg=<value optimized out>) at qpid/management/ManagementAgent.cpp:158 #16 0x00839bca in ~auto_ptr (this=0x8582230, __in_chrg=<value optimized out>) at /usr/include/c++/4.4.5/backward/auto_ptr.h:168 #17 qpid::broker::Broker::~Broker (this=0x8582230, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:405 #18 0x0083a403 in qpid::broker::Broker::~Broker (this=0x8582230, __in_chrg=<value optimized out>) at qpid/broker/Broker.cpp:405 #19 0x0082a866 in qpid::RefCounted::released (this=0x8582248) at qpid/RefCounted.h:48 #20 0x080547ec in release (this=0xbf80a05f, options=0x8579b20) at qpid/RefCounted.h:42 #21 intrusive_ptr_release (this=0xbf80a05f, options=0x8579b20) at qpid/RefCounted.h:57 #22 ~intrusive_ptr (this=0xbf80a05f, options=0x8579b20) at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101 #23 QpiddBroker::execute (this=0xbf80a05f, options=0x8579b20) at posix/QpiddBroker.cpp:187 #24 0x08050c81 in main (argc=-172655616, argv=0x2eff4ff) at qpidd.cpp:80 (gdb) quit
It's likely this is addressed by the change Ken made for bug 756446. Andrew, if you agree, you can set this to POST.
Seems to me very likely to be the same bug. setting to post.
Fixed upstream at http://svn.apache.org/viewvc?view=rev&rev=1207877
VERIFIED RHEL6.2 (i686/ x86_64) no coredump generated within 500runs rpm -qa | grep -P '(wallaby|qpid|sesame|condor|qmf)' | sort -u condor-7.6.5-0.12.el6.i686 condor-classads-7.6.5-0.12.el6.i686 condor-qmf-7.6.5-0.12.el6.i686 condor-wallaby-base-db-1.19-1.el6.noarch condor-wallaby-client-4.1.2-1.el6.noarch condor-wallaby-tools-4.1.2-1.el6.noarch python-condorutils-1.5-4.el6.noarch python-qpid-0.14-2.el6.noarch python-qpid-qmf-0.14-3.el6.i686 python-wallaby-0.12.5-1.el6.noarch python-wallabyclient-4.1.2-1.el6.noarch qpid-cpp-client-0.14-6.el6.i686 qpid-cpp-debuginfo-0.14-6.el6.i686 qpid-cpp-server-0.14-6.el6.i686 qpid-qmf-0.14-3.el6.i686 qpid-tools-0.14-1.el6.noarch ruby-qpid-qmf-0.14-3.el6.i686 ruby-wallaby-0.12.5-1.el6.noarch sesame-1.0-2.el6.i686 wallaby-0.12.5-1.el6.noarch wallaby-utils-0.12.5-1.el6.noarch i=0;while [[ $i -lt 500 ]] ; do service qpidd start; qpid-config add queue qname_test --file-count=64 --file-size=1 --durable --cluster-durable; qpid-config queues qname_test; kill -SIGINT $(pidof qpidd); sleep 4; ((i++)); echo ${i}; done
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2012-0529.html