Bug 692422

Summary: qpid c++ qmf client (qmf synchronous console) occasionally aborts with 'shared_ptr.hpp:253: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed.'
Product: Red Hat Enterprise MRG Reporter: Frantisek Reznicek <freznice>
Component: qpid-qmfAssignee: Ken Giusti <kgiusti>
Status: CLOSED WONTFIX QA Contact: MRG Quality Engineering <mrgqe-bugs>
Severity: high Docs Contact:
Priority: high    
Version: DevelopmentCC: esammons, iboverma, jross, kgiusti, matt
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-06-18 13:42:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Frantisek Reznicek 2011-03-31 10:17:06 UTC 
Description of problem:

Testing qpid qmf c++ console qmf_list_objects aborts occasionally with following boost assertion:
qmf_list_objects: /usr/include/boost/shared_ptr.hpp:253: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed.
Aborted (core dumped)

The c++ console [1] does not use any boost::shared_ptr elements:
[root@mrg-qe-05 c++]# grep shared_ptr qmf*.*pp
[root@mrg-qe-05 c++]# grep shared qmf*.*pp
[root@mrg-qe-05 c++]#

The issue might be


Version-Release number of selected component (if applicable):
[root@mrg-qe-05 c++]# rpm -qa | egrep '(qpid|sesame)' | sort
python-qpid-0.10-1.el5
python-qpid-qmf-0.10-2.el5
qpid-cpp-client-0.10-1.el5
qpid-cpp-client-devel-0.10-1.el5
qpid-cpp-server-0.10-1.el5
qpid-qmf-0.10-2.el5
qpid-qmf-devel-0.10-2.el5
qpid-tools-0.10-1.el5
ruby-qpid-qmf-0.10-2.el5


How reproducible:
10%

Steps to Reproduce:
1. set up MRG/M+G to have at least three QMF packages com.redhat.grid              org.apache.qpid.broker  com.redhat.grid.config
2. run the python qmf console in the loop [2]
cnt=0;while true; do ./qmf_list_objects.py --print-format=%p >l; [ "$(cat l | wc -l)" != "3" ] && echo -n "ERROR:"; echo "$(date +%Y%m%d_%H%M%S) ${cnt}: $(column l)" ; let "cnt++"; done;
3. run the c++ qmf console in the loop [1]
ulimit -c unlimited
while true; do ./qmf_list_objects || break; done;




References:
[1] http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/c++/
search for qmf_*

[2] http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/tests/distribution/MRG/Messaging/qpid_common/clients/python/
search for qmf_*



  
Actual results:
Qpid QMF c++ console sometimes aborts with boost::shared_ptr assertion.

Expected results:
Qpid QMF c++ console should not abort with boost::shared_ptr assertion.

Additional info:

[root@mrg-qe-05 c++]# ./qmf_list_objects --user guest --password guest
com.redhat.grid.config
  ConfigVersion
    1295341272561406 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929217] 1295341272561406)
      version   1295341272561406
      getNodeConfig(node, config)       No method description
    1295341612386827 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929218] 1295341612386827)
      version   1295341612386827
      getNodeConfig(node, config)       No method description
    1296642242562452 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929219] 1296642242562452)
      version   1296642242562452
      getNodeConfig(node, config)       No method description
...
    1301496465685155 (com.redhat.grid.config:ConfigVersion[0-0-1-71-4201078327361929260] 1301496465685155)
      version   1301496465685155
      getNodeConfig(node, config)       No method description
  Feature
qmf_list_objects: /usr/include/boost/shared_ptr.hpp:253: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed.
Aborted (core dumped)
[root@mrg-qe-05 c++]# file core.17235
core.17235: ELF 32-bit LSB core file Intel 80386, version 1 (SYSV), SVR4-style, from 'qmf_list_object'
[root@mrg-qe-05 c++]# gdb  ./qmf_list_objects core.17235
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-32.el5_6.2)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /root/c++/qmf_list_objects...done.

warning: core file may not match specified executable file.
[New Thread 17237]
[New Thread 17236]
...
Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libz.so.1
Core was generated by `./qmf_list_objects --user guest --password guest'.
Program terminated with signal 6, Aborted.
#0  0x007c9410 in __kernel_vsyscall ()
(gdb) info threads
  3 Thread 17236  0x007c9410 in __kernel_vsyscall ()
  2 Thread 17237  0x007c9410 in __kernel_vsyscall ()
* 1 Thread 17235  0x007c9410 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 3 (Thread 17236):
#0  0x007c9410 in __kernel_vsyscall ()
#1  0x00703bc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x0067677d in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libc.so.6
#3  0x00841dc3 in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop(qpid::sys::Duration) () from /usr/lib/libqpidclient.so.5
#4  0x0083e337 in qpid::client::Dispatcher::run() () from /usr/lib/libqpidclient.so.5
#5  0x00870098 in qpid::client::SubscriptionManagerImpl::run() () from /usr/lib/libqpidclient.so.5
#6  0x0086e0c4 in qpid::client::SubscriptionManager::run() () from /usr/lib/libqpidclient.so.5
#7  0x00c29b7a in qpid::console::Broker::ConnectionThread::run() () from /usr/lib/libqmfconsole.so.5
#8  0x00e2d341 in ?? () from /usr/lib/libqpidcommon.so.5
#9  0x006ff832 in start_thread () from /lib/libpthread.so.0
#10 0x0066a0ae in clone () from /lib/libc.so.6

Thread 2 (Thread 17237):
#0  0x007c9410 in __kernel_vsyscall ()
#1  0x0066a726 in epoll_wait () from /lib/libc.so.6
#2  0x00e3726a in qpid::sys::Poller::wait(qpid::sys::Duration) () from /usr/lib/libqpidcommon.so.5
#3  0x00e37e93 in qpid::sys::Poller::run() () from /usr/lib/libqpidcommon.so.5
#4  0x00e2d341 in ?? () from /usr/lib/libqpidcommon.so.5
#5  0x006ff832 in start_thread () from /lib/libpthread.so.0
#6  0x0066a0ae in clone () from /lib/libc.so.6

Thread 1 (Thread 17235):
#0  0x007c9410 in __kernel_vsyscall ()
#1  0x005c0df0 in raise () from /lib/libc.so.6
#2  0x005c2701 in abort () from /lib/libc.so.6
#3  0x005ba26b in __assert_fail () from /lib/libc.so.6
#4  0x08067e0f in boost::shared_ptr<qpid::console::Value>::operator-> (this=0xb6a2f164)
    at /usr/include/boost/shared_ptr.hpp:253
#5  0x080640b4 in qmf_lib::get_qmfobj_name (in_obj=...) at qmf_console_reader.cpp:329
#6  0x0806dfef in main_int (opts=...) at qmf_list_objects.cpp:544
#7  0x08071850 in main (argc=5, argv=0xbfbab684) at qmf_list_objects.cpp:772
(gdb) quit
[root@mrg-qe-05 c++]# head -1 /etc/issue
Red Hat Enterprise Linux Server release 5.6 (Tikanga)
[root@mrg-qe-05 c++]# uname -a
Linux mrg-qe-05.lab.eng.brq.redhat.com 2.6.18-238.1.1.el5PAE #1 SMP Tue Jan 4 13:53:16 EST 2011 i686 athlon i386 GNU/Linux
Comment 1 Frantisek Reznicek 2011-03-31 10:27:20 UTC 
After couple of trials I was able to get the same behavior on different boost version as well and found the troublemaker...

There seems to be ininitialized boost::shared_ptr around
qpid::console::Object::AttributeMap iterator access to map of qmf object attributes:

  // following access sometimes causes the above boost assertion
  it->second->str().length();
  // where it is bject::AttributeMap::const_iterator


see details below:

[freznice@dhcp-26-251 c++]$ ./qmf_list_objects --user guest --password guest
  vhost
    / (org.apache.qpid.broker:vhost[0-0-1-0-3] 0-0-1-0-2:/)
      brokerRef 0-0-1-0-2
      federationTag     ea442b19-ba2f-4ac5-9ddb-fe2f35fb000d
      name      /
com.redhat.grid.config
  Group
qmf_list_objects: /usr/include/boost/shared_ptr.hpp:315: T* boost::shared_ptr<T>::operator->() const [with T = qpid::console::Value]: Assertion `px != 0' failed.
Aborted (core dumped)
[freznice@dhcp-26-251 c++]$ gdb ./qmf_list_objects core.10                                           core.10062  core.10155  core.10234  core.10315  core.10411  core.10509  core.10695
[freznice@dhcp-26-251 c++]$ gdb ./qmf_list_objects core.10695
...
Loaded symbols for /usr/lib64/sasl2/libplain.so.2.0.22
Core was generated by `./qmf_list_objects -b mrg-qe-05 -u guest --password guest'.
Program terminated with signal 6, Aborted.
[New process 10695]
[New process 10697]
[New process 10696]
#0  0x0000003d73630ec5 in raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install keyutils.x86_64 libselinux.x86_64 mysql.x86_64 postgresql.x86_64
(gdb) info threads
  3 process 10696  0x0000003d7420a8f9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread-2.7.so
  2 process 10697  0x0000003d736d50d8 in epoll_wait () from /lib64/libc-2.7.so
* 1 process 10695  0x0000003d73630ec5 in raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
Current language:  auto; currently c
(gdb) thread apply all bt

Thread 3 (process 10696):
#0  0x0000003d7420a8f9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread-2.7.so
#1  0x00000032fe078a8b in qpid::sys::BlockingQueue<boost::shared_ptr<qpid::framing::FrameSet> >::pop
    (this=<value optimized out>, timeout=<value optimized out>)
    at ../include/qpid/sys/posix/Condition.h:63
#2  0x00000032fe0756d7 in qpid::client::Dispatcher::run (this=<value optimized out>)
    at qpid/client/Dispatcher.cpp:80
#3  0x000000000067762f in qpid::console::Broker::ConnectionThread::run (this=0x255ae50)
    at qpid/console/Broker.cpp:203
#4  0x00000032fd72656a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#5  0x0000003d74206407 in start_thread (arg=<value optimized out>) at pthread_create.c:297
#6  0x0000003d736d4b0d in clone () from /lib64/libc-2.7.so

Thread 2 (process 10697):
#0  0x0000003d736d50d8 in epoll_wait () from /lib64/libc-2.7.so
#1  0x00000032fd72f841 in qpid::sys::Poller::wait (this=<value optimized out>,
    timeout=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:563
#2  0x00000032fd7302b7 in qpid::sys::Poller::run (this=<value optimized out>)
    at qpid/sys/epoll/EpollPoller.cpp:515
#3  0x00000032fd72656a in runRunnable (p=<value optimized out>) at qpid/sys/posix/Thread.cpp:35
#4  0x0000003d74206407 in start_thread (arg=<value optimized out>) at pthread_create.c:297
#5  0x0000003d736d4b0d in clone () from /lib64/libc-2.7.so
Current language:  auto; currently asm

Thread 1 (process 10695):
#0  0x0000003d73630ec5 in raise (sig=<value optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x0000003d73632970 in abort () at abort.c:88
#2  0x0000003d7362a11f in __assert_fail (assertion=<value optimized out>,
    file=<value optimized out>, line=<value optimized out>, function=<value optimized out>)
    at assert.c:78
#3  0x000000000042bfd3 in boost::shared_ptr<qpid::console::Value>::operator-> (this=0x25ea8a8)
    at /usr/include/boost/shared_ptr.hpp:315
#4  0x000000000042942f in qmf_lib::get_qmfobj_name (in_obj=@0x261b530) at qmf_console_reader.cpp:331
#5  0x0000000000431a62 in main_int (opts=@0x7fff0b8cca30) at qmf_list_objects.cpp:544
#6  0x0000000000434c4d in main (argc=7, argv=0x7fff0b8ccce8) at qmf_list_objects.cpp:772
Current language:  auto; currently c
(gdb) quit


Modified source code: (qmf_console_reader.cpp)

    const Object::AttributeMap& int_attrs = in_obj.getAttributes();
    for (Object::AttributeMap::const_iterator it = int_attrs.begin();
        it != int_attrs.end(); it++)
    {

 ...
 
 331  int len1=it->second->str().length();
 332  int len2=int_id.length();
   3  if ((int_id.length()-1-it->second->str().length())>=0)
   4  {
   5    if (int_id.find( it->second->str(),
   6             int_id.length()-1-it->second->str().length() ) != string::npos)
   7    {
   8      // found name contains substr (at the end) of in_obj.getIndex()
   9      if (it->second->str().length() > int_sname.length())
 340        int_sname = it->second->str();
   1    }
   2  }