Created attachment 409444 [details] The testing client (g++ -Wall -g -I. -I/usr/include/qpid-boost bz452015.cpp -lpthread -lqpidclient -o bz452015) Description of problem: There were observed multiple crashes: a] directly in qpid::loadModuleDir() call (rhel5.5 case) Core was generated by `./bz452015'. Program terminated with signal 11, Segmentation fault. #0 qpid::loadModuleDir (dirname=..., isDefault=true) at /usr/include/c++/4.1.2/bits/basic_string.h:1563 1563 { return this->find(__str.data(), __pos, __str.size()); } (gdb) info threads * 1 Thread 31557 qpid::loadModuleDir (dirname=..., isDefault=true) at /usr/include/c++/4.1.2/bits/basic_string.h:1563 (gdb) bt #0 qpid::loadModuleDir (dirname=..., isDefault=true) at /usr/include/c++/4.1.2/bits/basic_string.h:1563 #1 0x007e94f2 in LoadtimeInitialise (__initialize_p=<value optimized out>, __priority=<value optimized out>) at qpid/client/LoadPlugins.cpp:47 #2 __static_initialization_and_destruction_0 (__initialize_p=<value optimized out>, __priority=<value optimized out>) at qpid/client/LoadPlugins.cpp:50 #3 0x00872cb6 in __do_global_ctors_aux () from /usr/lib/libqpidclient.so.2 #4 0x00796d95 in _init () from /usr/lib/libqpidclient.so.2 #5 0x00217223 in call_init () from /lib/ld-linux.so.2 #6 0x00217333 in _dl_init_internal () from /lib/ld-linux.so.2 #7 0x0020984f in _dl_start_user () from /lib/ld-linux.so.2 b] or indirectly in ::isShlibName() called by qpid::loadModuleDir() (rhel4.8 case) Core was generated by `./bz452015'. Program terminated with signal 11, Segmentation fault. ... #0 (anonymous namespace)::isShlibName (name=@0x8247840) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/basic_string.h:278 278 { return &((reinterpret_cast<_Rep*> (_M_data()))[-1]); } (gdb) info threads * 1 process 23742 (anonymous namespace)::isShlibName (name=@0x8247840) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/basic_string.h:278 (gdb) bt #0 (anonymous namespace)::isShlibName (name=@0x8247840) at /usr/lib/gcc/i386-redhat-linux/3.4.6/../../../../include/c++/3.4.6/bits/basic_string.h:278 #1 0x004ad7ef in qpid::loadModuleDir (dirname= {static npos = 4294967295, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0x824823c "/usr/lib/qpid/client"}}, isDefault=true) at /usr/include/boost/filesystem/path.hpp:644 #2 0x007ccfdb in global constructors keyed to _ZN56_GLOBAL__N_qpid_client_LoadPlugins.cpp_9D2164D3_2299BD904initE () at qpid/client/LoadPlugins.cpp:47 #3 0x0088bd95 in __do_global_ctors_aux () from /usr/lib/libqpidclient.so.2 #4 0x0076fa01 in _init () from /usr/lib/libqpidclient.so.2 #5 0x00ade8e8 in _dl_init_internal () from /lib/ld-linux.so.2 #6 0x00ad27ff in _dl_start_user () from /lib/ld-linux.so.2 Version-Release number of selected component (if applicable): qpid-java-common-0.7.934605-1.el5 qpid-cpp-client-ssl-0.7.935473-1.el5 qpid-cpp-server-store-0.7.935473-1.el5 qpid-cpp-client-devel-docs-0.7.935473-1.el5 qpid-tests-0.7.930108-1.el5 python-qpid-0.7.934605-1.el5 qpid-tools-0.7.934605-2.el5 qmf-devel-0.7.935473-1.el5 qpid-java-client-0.7.934605-1.el5 qpid-cpp-server-0.7.935473-1.el5 qpid-cpp-client-devel-0.7.935473-1.el5 qpid-cpp-server-devel-0.7.935473-1.el5 qpid-cpp-server-cluster-0.7.935473-1.el5 ruby-qpid-0.7.935473-1.el5 qpid-cpp-client-0.7.935473-1.el5 qpid-cpp-server-xml-0.7.935473-1.el5 qpid-cpp-mrg-debuginfo-0.7.935473-1.el5 python-qmf-0.7.934605-1.el5 qmf-0.7.935473-1.el5 qpid-cpp-server-ssl-0.7.935473-1.el5 How reproducible: >90% (very rapidly with attached client) Steps to Reproduce: 1. start qpidd broker 2. 'qpid-config add exchange direct bz452015' 3. './bz452015' 4. expect failure but not crash 5. 'qpid-config del exchange bz452015' 6. 'qpid-config --sequence add exchange direct bz452015' 7. './bz452015' 8. expect PASS Actual results: Attached client crashes at steps 3. and 7. Expected results: Attached client should not crash. Additional info:
The issue here comes from linking in pthreads before qpidclient. If you compile the client with: g++ -Wall -g -I. -I/usr/include/qpid-boost bz452015.cpp -lqpidclient -o bz452015, then the above expectations are met. In fact even if the -lpthread is added in after the -lqpidclient then the issue is resolved. A similar issue can be seen if you link in pthread before any qpid client based executable (e.g. direct exchange example).
Yes, indeed, the crash is caused by linking libpthread before libqpidclient. Clients behave correctly now.
The cause of this crash is undefined initialisation order: LoadtimeInitialise() at qpid/client/LoadPlugins.cpp:47 which is part of a static global constructor and so is called before main(). It relies transitively on suffix in qpid/Modules.cpp:39 which is also a static global also constructed before main(). These things are in different files nad in fact in entirely different shared objects. Since c++ does not define the order of construction of static globals (except in the same file) under some circumstances the construction order may work because suffix gets constructed before LoadtimeInitialise(), and in others it may not thus causing this crash.
This should now be fixed upstream at r938701 I fixed it with the "belt and braces" approach of making LoadtimeInitialise no longer run as part of a static global constructor and also making suffix accessed using a "static singleton" pattern.
The issue has been fixed, tested on RHEL 4.8 / 5.5 i386 / x86_64 on packages: python-qmf-0.7.946106-7.el5 python-qpid-0.7.946106-9.el5 qmf-0.7.946106-10.el5 qmf-devel-0.7.946106-10.el5 qpid-cpp-client*-0.7.946106-10.el5 qpid-cpp-mrg-debuginfo-0.7.946106-8.el5 qpid-cpp-server*-0.7.946106-10.el5 qpid-java-*-0.7.946106-7.el5 qpid-tools-0.7.946106-7.el5 ruby-qmf-0.7.946106-10.el5 -> VERIFIED