Hide Forgot
Description of problem: Testing qpid qmf c++ console qmf_list_objects aborts occasionally with following boost assertion: qmf_list_objects: /usr/include/boost/shared_ptr.hpp:253: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed. Aborted (core dumped) The c++ console [1] does not use any boost::shared_ptr elements: [root@mrg-qe-05 c++]# grep shared_ptr qmf*.*pp [root@mrg-qe-05 c++]# grep shared qmf*.*pp [root@mrg-qe-05 c++]# The issue might be Version-Release number of selected component (if applicable): [root@mrg-qe-05 c++]# rpm -qa | egrep '(qpid|sesame)' | sort python-qpid-0.10-1.el5 python-qpid-qmf-0.10-2.el5 qpid-cpp-client-0.10-1.el5 qpid-cpp-client-devel-0.10-1.el5 qpid-cpp-server-0.10-1.el5 qpid-qmf-0.10-2.el5 qpid-qmf-devel-0.10-2.el5 qpid-tools-0.10-1.el5 ruby-qpid-qmf-0.10-2.el5 How reproducible: 10% Steps to Reproduce: 1. set up MRG/M+G to have at least three QMF packages com.redhat.grid org.apache.qpid.broker com.redhat.grid.config 2. run the python qmf console in the loop [2] cnt=0;while true; do ./qmf_list_objects.py --print-format=%p >l; [ "$(cat l | wc -l)" != "3" ] && echo -n "ERROR:"; echo "$(date +%Y%m%d_%H%M%S) ${cnt}: $(column l)" ; let "cnt++"; done; 3. run the c++ qmf console in the loop [1] ulimit -c unlimited while true; do ./qmf_list_objects || break; done; References: [1] http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/c++/ search for qmf_* [2] http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/python/ search for qmf_* Actual results: Qpid QMF c++ console sometimes aborts with boost::shared_ptr assertion. Expected results: Qpid QMF c++ console should not abort with boost::shared_ptr assertion. Additional info: [root@mrg-qe-05 c++]# ./qmf_list_objects --user guest --password guest com.redhat.grid.config ConfigVersion 1295341272561406 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929217] 1295341272561406) version 1295341272561406 getNodeConfig(node, config) No method description 1295341612386827 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929218] 1295341612386827) version 1295341612386827 getNodeConfig(node, config) No method description 1296642242562452 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929219] 1296642242562452) version 1296642242562452 getNodeConfig(node, config) No method description ... 1301496465685155 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929260] 1301496465685155) version 1301496465685155 getNodeConfig(node, config) No method description Feature qmf_list_objects: /usr/include/boost/shared_ptr.hpp:253: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed. Aborted (core dumped) [root@mrg-qe-05 c++]# file core.17235 core.17235: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qmf_list_object' [root@mrg-qe-05 c++]# gdb ./qmf_list_objects core.17235 GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /root/c++/qmf_list_objects...done. warning: core file may not match specified executable file. [New Thread 17237] [New Thread 17236] ... Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Core was generated by `./qmf_list_objects --user guest --password guest'. Program terminated with signal 6, Aborted. #0 0x007c9410 in __kernel_vsyscall () (gdb) info threads 3 Thread 17236 0x007c9410 in __kernel_vsyscall () 2 Thread 17237 0x007c9410 in __kernel_vsyscall () * 1 Thread 17235 0x007c9410 in __kernel_vsyscall () (gdb) thread apply all bt Thread 3 (Thread 17236): #0 0x007c9410 in __kernel_vsyscall () #1 0x00703bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0 #2 0x0067677d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6 #3 0x00841dc3 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop(qpid::sys::Duration) () from /usr/lib/libqpidclient.so.5 #4 0x0083e337 in qpid::client::Dispatcher::run() () from /usr/lib/libqpidclient.so.5 #5 0x00870098 in qpid::client::SubscriptionManagerImpl::run() () from /usr/lib/libqpidclient.so.5 #6 0x0086e0c4 in qpid::client::SubscriptionManager::run() () from /usr/lib/libqpidclient.so.5 #7 0x00c29b7a in qpid::console::Broker::ConnectionThread::run() () from /usr/lib/libqmfconsole.so.5 #8 0x00e2d341 in ?? () from /usr/lib/libqpidcommon.so.5 #9 0x006ff832 in start_thread () from /lib/libpthread.so.0 #10 0x0066a0ae in clone () from /lib/libc.so.6 Thread 2 (Thread 17237): #0 0x007c9410 in __kernel_vsyscall () #1 0x0066a726 in epoll_wait () from /lib/libc.so.6 #2 0x00e3726a in qpid::sys::Poller::wait(qpid::sys::Duration) () from /usr/lib/libqpidcommon.so.5 #3 0x00e37e93 in qpid::sys::Poller::run() () from /usr/lib/libqpidcommon.so.5 #4 0x00e2d341 in ?? () from /usr/lib/libqpidcommon.so.5 #5 0x006ff832 in start_thread () from /lib/libpthread.so.0 #6 0x0066a0ae in clone () from /lib/libc.so.6 Thread 1 (Thread 17235): #0 0x007c9410 in __kernel_vsyscall () #1 0x005c0df0 in raise () from /lib/libc.so.6 #2 0x005c2701 in abort () from /lib/libc.so.6 #3 0x005ba26b in __assert_fail () from /lib/libc.so.6 #4 0x08067e0f in boost::shared_ptr<qpid::console::Value>::operator-> (this=0xb6a2f164) at /usr/include/boost/shared_ptr.hpp:253 #5 0x080640b4 in qmf_lib::get_qmfobj_name (in_obj=...) at qmf_console_reader.cpp:329 #6 0x0806dfef in main_int (opts=...) at qmf_list_objects.cpp:544 #7 0x08071850 in main (argc=5, argv=0xbfbab684) at qmf_list_objects.cpp:772 (gdb) quit [root@mrg-qe-05 c++]# head -1 /etc/issue Red Hat Enterprise Linux Server release 5.6 (Tikanga) [root@mrg-qe-05 c++]# uname -a Linux mrg-qe-05.lab.eng.brq.redhat.com 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 athlon i386 GNU/Linux
After couple of trials I was able to get the same behavior on different boost version as well and found the troublemaker... There seems to be ininitialized boost::shared_ptr around qpid::console::Object::AttributeMap iterator access to map of qmf object attributes: // following access sometimes causes the above boost assertion it->second->str().length(); // where it is bject::AttributeMap::const_iterator see details below: [freznice@dhcp-26-251 c++]$ ./qmf_list_objects --user guest --password guest vhost / (org.apache.qpid.broker:vhost[0-0-1-0-3] 0-0-1-0-2:/) brokerRef 0-0-1-0-2 federationTag ea442b19-ba2f-4ac5-9ddb-fe2f35fb000d name / com.redhat.grid.config Group qmf_list_objects: /usr/include/boost/shared_ptr.hpp:315: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed. Aborted (core dumped) [freznice@dhcp-26-251 c++]$ gdb ./qmf_list_objects core.10 core.10062 core.10155 core.10234 core.10315 core.10411 core.10509 core.10695 [freznice@dhcp-26-251 c++]$ gdb ./qmf_list_objects core.10695 ... Loaded symbols for /usr/lib64/sasl2/libplain.so.2.0.22 Core was generated by `./qmf_list_objects -b mrg-qe-05 -u guest --password guest'. Program terminated with signal 6, Aborted. [New process 10695] [New process 10697] [New process 10696] #0 0x0000003d73630ec5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Missing separate debuginfos, use: debuginfo-install keyutils.x86_64 libselinux.x86_64 mysql.x86_64 postgresql.x86_64 (gdb) info threads 3 process 10696 0x0000003d7420a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread-2.7.so 2 process 10697 0x0000003d736d50d8 in epoll_wait () from /lib64/libc-2.7.so * 1 process 10695 0x0000003d73630ec5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 Current language: auto; currently c (gdb) thread apply all bt Thread 3 (process 10696): #0 0x0000003d7420a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread-2.7.so #1 0x00000032fe078a8b in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop (this=<value optimized out>, timeout=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:63 #2 0x00000032fe0756d7 in qpid::client::Dispatcher::run (this=<value optimized out>) at qpid/client/Dispatcher.cpp:80 #3 0x000000000067762f in qpid::console::Broker::ConnectionThread::run (this=0x255ae50) at qpid/console/Broker.cpp:203 #4 0x00000032fd72656a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #5 0x0000003d74206407 in start_thread (arg=<value optimized out>) at pthread_create.c:297 #6 0x0000003d736d4b0d in clone () from /lib64/libc-2.7.so Thread 2 (process 10697): #0 0x0000003d736d50d8 in epoll_wait () from /lib64/libc-2.7.so #1 0x00000032fd72f841 in qpid::sys::Poller::wait (this=<value optimized out>, timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563 #2 0x00000032fd7302b7 in qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:515 #3 0x00000032fd72656a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35 #4 0x0000003d74206407 in start_thread (arg=<value optimized out>) at pthread_create.c:297 #5 0x0000003d736d4b0d in clone () from /lib64/libc-2.7.so Current language: auto; currently asm Thread 1 (process 10695): #0 0x0000003d73630ec5 in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x0000003d73632970 in abort () at abort.c:88 #2 0x0000003d7362a11f in __assert_fail (assertion=<value optimized out>, file=<value optimized out>, line=<value optimized out>, function=<value optimized out>) at assert.c:78 #3 0x000000000042bfd3 in boost::shared_ptr<qpid::console::Value>::operator-> (this=0x25ea8a8) at /usr/include/boost/shared_ptr.hpp:315 #4 0x000000000042942f in qmf_lib::get_qmfobj_name (in_obj=@0x261b530) at qmf_console_reader.cpp:331 #5 0x0000000000431a62 in main_int (opts=@0x7fff0b8cca30) at qmf_list_objects.cpp:544 #6 0x0000000000434c4d in main (argc=7, argv=0x7fff0b8ccce8) at qmf_list_objects.cpp:772 Current language: auto; currently c (gdb) quit Modified source code: (qmf_console_reader.cpp) const Object::AttributeMap& int_attrs = in_obj.getAttributes(); for (Object::AttributeMap::const_iterator it = int_attrs.begin(); it != int_attrs.end(); it++) { ... 331 int len1=it->second->str().length(); 332 int len2=int_id.length(); 3 if ((int_id.length()-1-it->second->str().length())>=0) 4 { 5 if (int_id.find( it->second->str(), 6 int_id.length()-1-it->second->str().length() ) != string::npos) 7 { 8 // found name contains substr (at the end) of in_obj.getIndex() 9 if (it->second->str().length() > int_sname.length()) 340 int_sname = it->second->str(); 1 } 2 }