Bug 582366

Summary: When reloading a large acl file , the broker core dumps
Product: Red Hat Enterprise MRG Reporter: Rajith Attapattu <rattapat+nobody>
Component: qpid-cppAssignee: Rajith Attapattu <rattapat+nobody>
Status: CLOSED ERRATA QA Contact: ppecka <ppecka>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.2CC: freznice, gsim, ppecka
Target Milestone: 1.3   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
When reloading a large 'acl' file, the broker core was dumped. With this update, the broker continues to work after reloading the 'acl' file as expected.
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-14 16:01:26 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 568718    
Attachments:
Description Flags
Reproducer none

Description Rajith Attapattu 2010-04-14 18:25:10 UTC
Description of problem:
When reloading a large acl file , the broker core dumps.
This surfaced while running a test program to validate bz 568718

Version-Release number of selected component (if applicable):
Affects versions btw 1.2 - 1.3

How reproducible:
Always -  reproducer attached.

Steps to Reproduce:
1. Start the broker with the acl module and and --acl-file /tmp/policy.acl
2. The initial acl file should contain only "acl allow all all"
3. run the message_sender.py (This program will keep on publishing to amq.direct)
4. run acl_reloader.py with --mode allow | deny a few times
  
Actual results:
The broker core dumps.

Expected results:
The broker should continue to work after reloading the acl file properly.

Additional info:

Following is the backtrace from the code dump.

   1.
      (gdb) bt
   2.
      #0  0x00cbe422 in __kernel_vsyscall ()
   3.
      #1  0x00183781 in raise () from /lib/libc.so.6
   4.
      #2  0x0018504a in abort () from /lib/libc.so.6
   5.
      #3  0x001c1619 in __libc_message () from /lib/libc.so.6
   6.
      #4  0x001c7a71 in malloc_printerr () from /lib/libc.so.6
   7.
      #5  0x001ca363 in munmap_chunk () from /lib/libc.so.6
   8.
      #6  0x040a1681 in operator delete(void*) () from /usr/lib/libstdc++.so.6
   9.
      #7  0x0035243e in qpid::acl::AclData::clear (this=0x8221328) at qpid/acl/AclData.cpp:40
  10.
      #8  0x003524ad in qpid::acl::AclData::~AclData (this=0x8221328, __in_chrg=<value optimized out>) at qpid/acl/AclData.cpp:259
  11.
      #9  0x003515a8 in checked_delete<qpid::acl::AclData> (x=<value optimized out>) at /usr/include/boost/checked_delete.hpp:34
  12.
      #10 boost::detail::sp_counted_impl_p<qpid::acl::AclData>::dispose (x=<value optimized out>) at /usr/include/boost/detail/sp_counted_impl.hpp:78
  13.
      #11 0x0034e50b in boost::detail::sp_counted_base::release (this=<value optimized out>) at /usr/include/boost/detail/sp_counted_base_gcc_x86.hpp:145
  14.
      #12 ~shared_count (this=<value optimized out>) at /usr/include/boost/detail/shared_count.hpp:216
  15.
      #13 ~shared_ptr (this=<value optimized out>) at /usr/include/boost/shared_ptr.hpp:165
  16.
      #14 qpid::acl::Acl::authorise (this=<value optimized out>) at qpid/acl/Acl.cpp:86
  17.
      #15 0x00add720 in qpid::broker::SemanticState::route (this=0x82218a0, msg={p_ = 0xb5644868}, strategy=@0xb61fe178)
  18.
          at qpid/broker/SemanticState.cpp:447
  19.
      #16 0x00ade215 in qpid::broker::SemanticState::handle (this=0x82218a0, msg={p_ = 0xb5644868}) at qpid/broker/SemanticState.cpp:415
  20.
      #17 0x00b03c50 in qpid::broker::SessionState::handleContent (this=0x8221778, frame=@0xb61feb00, id=@0xb61fe3c8) at qpid/broker/SessionState.cpp:249
  21.
      #18 0x00b04271 in qpid::broker::SessionState::handleIn (this=0x8221778, frame=@0xb61feb00) at qpid/broker/SessionState.cpp:327
  22.
      #19 0x00b046fb in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x8221878,
  23.
          t=@0xb61feb00) at ./qpid/framing/Handler.h:67
  24.
      #20 0x005156f2 in qpid::amqp_0_10::SessionHandler::handleIn (this=0x827ddc8, f=@0xb61feb00) at qpid/amqp_0_10/SessionHandler.cpp:93
  25.
      #21 0x00b046fb in qpid::framing::Handler<qpid::framing::AMQFrame&>::MemFunRef<qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface, &(qpid::framing::Handler<qpid::framing::AMQFrame&>::InOutHandlerInterface::handleIn(qpid::framing::AMQFrame&))>::handle (this=0x827ddd0,
  26.
          t=@0xb61feb00) at ./qpid/framing/Handler.h:67
  27.
      #22 0x00a58f50 in qpid::framing::Handler<qpid::framing::AMQFrame&>::operator() (t=<value optimized out>, this=<value optimized out>)
  28.
          at ./qpid/framing/Handler.h:42
  29.
      #23 qpid::broker::Connection::received (t=<value optimized out>, this=<value optimized out>) at qpid/broker/Connection.cpp:143
  30.
      #24 0x00a39b63 in qpid::amqp_0_10::Connection::decode (this=0x82265f0, buffer=0x82adfd0 "\v\1", size=82) at qpid/amqp_0_10/Connection.cpp:58
  31.
      #25 0x00ad98ab in qpid::broker::SecureConnection::decode (this=0x81f9988, buffer=0x6 <Address 0x6 out of bounds>, size=6485)
  32.
          at qpid/broker/SecureConnection.cpp:42
  33.
      #26 0x0055e96d in qpid::sys::AsynchIOHandler::readbuff (this=0x81f7598, buff=0x81f7188) at qpid/sys/AsynchIOHandler.cpp:135
  34.
      #27 0x00b3f7ab in boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::operator() (
  35.
          a2=<value optimized out>, a1=<value optimized out>, p=<value optimized out>, this=<value optimized out>)
  36.
      ---Type <return> to continue, or q <return> to quit---
  37.
          at /usr/include/boost/bind/mem_fn_template.hpp:274
  38.
      #28 operator()<boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list2<qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*&> > (a2=<value optimized out>, a1=<value optimized out>, p=<value optimized out>,
  39.
          this=<value optimized out>) at /usr/include/boost/bind.hpp:371
  40.
      #29 operator()<qpid::sys::AsynchIO, qpid::sys::AsynchIOBufferBase*> (a2=<value optimized out>, a1=<value optimized out>, p=<value optimized out>,
  41.
          this=<value optimized out>) at /usr/include/boost/bind/bind_template.hpp:61
  42.
      #30 boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, qpid::sys::AsynchIOHandler, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>, boost::_bi::list3<boost::_bi::value<qpid::sys::AsynchIOHandler*>, boost::arg<1>, boost::arg<2> > >, void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::invoke (a2=<value optimized out>, a1=<value optimized out>, p=<value optimized out>,
  43.
          this=<value optimized out>) at /usr/include/boost/function/function_template.hpp:152
  44.
      #31 0x0047553a in boost::function2<void, qpid::sys::AsynchIO&, qpid::sys::AsynchIOBufferBase*>::operator() (this=0x82264f8, a0=@0x8226448,
  45.
          a1=0x81f7188) at /usr/include/boost/function/function_template.hpp:989
  46.
      #32 0x004731d5 in qpid::sys::posix::AsynchIO::readable (this=0x8226448, h=@0x822644c) at qpid/sys/posix/AsynchIO.cpp:418
  47.
      #33 0x004746a4 in boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>::operator() (a1=<value optimized out>,
  48.
          p=<value optimized out>, this=<value optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:162
  49.
      #34 operator()<boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list1<qpid::sys::DispatchHandle&> > (
  50.
          a1=<value optimized out>, p=<value optimized out>, this=<value optimized out>) at /usr/include/boost/bind.hpp:292
  51.
      #35 operator()<qpid::sys::DispatchHandle> (a1=<value optimized out>, p=<value optimized out>, this=<value optimized out>)
  52.
          at /usr/include/boost/bind/bind_template.hpp:32
  53.
      #36 boost::detail::function::void_function_obj_invoker1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, qpid::sys::posix::AsynchIO, qpid::sys::DispatchHandle&>, boost::_bi::list2<boost::_bi::value<qpid::sys::posix::AsynchIO*>, boost::arg<1> > >, void, qpid::sys::DispatchHandle&>::invoke (
  54.
          a1=<value optimized out>, p=<value optimized out>, this=<value optimized out>) at /usr/include/boost/function/function_template.hpp:152
  55.
      #37 0x00565d73 in boost::function1<void, qpid::sys::DispatchHandle&>::operator() (this=0x8226454, a0=@0x822644c)
  56.
          at /usr/include/boost/function/function_template.hpp:989
  57.
      #38 0x00562ccf in qpid::sys::DispatchHandle::processEvent (this=0x822644c, type=qpid::sys::Poller::READABLE) at qpid/sys/DispatchHandle.cpp:278
  58.
      #39 0x004827b3 in qpid::sys::Poller::Event::process (this=<value optimized out>) at ./qpid/sys/Poller.h:123
  59.
      #40 qpid::sys::Poller::run (this=<value optimized out>) at qpid/sys/epoll/EpollPoller.cpp:483
  60.
      #41 0x00566214 in qpid::sys::Dispatcher::run (this=0xbfcad1a8) at qpid/sys/Dispatcher.cpp:37
  61.
      #42 0x00477dc1 in qpid::sys::(anonymous namespace)::runRunnable (p=0xbfcad1a8) at qpid/sys/posix/Thread.cpp:35
  62.
      #43 0x00dc28f5 in start_thread () from /lib/libpthread.so.0
  63.
      #44 0x00235fce in clone () from /lib/libc.so.6

Comment 1 Rajith Attapattu 2010-04-14 18:37:49 UTC
Created attachment 406601 [details]
Reproducer

This issue is likely to happen with a sufficiently large acl file. (~ 1000+  entries)
With a smaller file (~100 entries) it takes a few iterations to happen.

Comment 2 Rajith Attapattu 2010-04-23 01:02:31 UTC
A fix for this has been committed at rev 937120 in Qpid trunk.
I will be adding a test case for this based on the reproducer.

Comment 3 ppecka 2010-05-31 12:50:17 UTC
verified on RHEL 5.5/4.8 - i386/x86_64:

rpm -qa | grep -E '(qpid|ais|sesame)' | sort -n
openais-0.80.6-16.el5_5.1
openais-debuginfo-0.80.6-16.el5_5.1
openais-devel-0.80.6-16.el5_5.1
python-qpid-0.7.946106-1.el5
qpid-cpp-client-0.7.946106-1.el5
qpid-cpp-client-devel-0.7.946106-1.el5
qpid-cpp-client-devel-docs-0.7.946106-1.el5
qpid-cpp-client-ssl-0.7.946106-1.el5
qpid-cpp-server-0.7.946106-1.el5
qpid-cpp-server-cluster-0.7.946106-1.el5
qpid-cpp-server-devel-0.7.946106-1.el5
qpid-cpp-server-ssl-0.7.946106-1.el5
qpid-cpp-server-store-0.7.946106-1.el5
qpid-cpp-server-xml-0.7.946106-1.el5
qpid-java-client-0.7.946106-3.el5
qpid-java-common-0.7.946106-3.el5
qpid-tools-0.7.946106-4.el5
rh-tests-distribution-MRG-Messaging-qpid_common-1.6-27
sesame-0.7.3918-2.el5

--> VERIFIED

Comment 4 Martin Prpič 2010-10-10 09:39:49 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
When reloading a large 'acl' file, the broker core was dumped. With this update, the broker continues to work after reloading the 'acl' file as expected.

Comment 6 errata-xmlrpc 2010-10-14 16:01:26 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0773.html