Hide Forgot
Description of problem: Qpidd broker ocassionly aborts during shutdown when txtest running. The broker shutdown was done by kill -2 <pid> and after some time kill -15 <pid> because qpidd did not start. This behavior was observed once on RHEL 5.6 i386. Version-Release number of selected component (if applicable): > python-qpid-0.10-1.el5 > python-qpid-qmf-0.10-2.el5 > python-saslwrapper-0.1.934605-2.el5 > qpid-cpp-client-0.10-1.el5 > qpid-cpp-client-devel-0.10-1.el5 > qpid-cpp-client-devel-docs-0.10-1.el5 > qpid-cpp-client-ssl-0.10-1.el5 > qpid-cpp-mrg-debuginfo-0.10-1.el5 > qpid-cpp-server-0.10-1.el5 > qpid-cpp-server-cluster-0.10-1.el5 > qpid-cpp-server-devel-0.10-1.el5 > qpid-cpp-server-ssl-0.10-1.el5 > qpid-cpp-server-store-0.10-1.el5 > qpid-cpp-server-xml-0.10-1.el5 > qpid-java-client-0.10-1.el5 > qpid-java-common-0.10-1.el5 > qpid-java-example-0.10-1.el5 > qpid-qmf-0.10-2.el5 > qpid-qmf-devel-0.10-2.el5 > qpid-tests-0.10-1.el5 > qpid-tools-0.10-1.el5 > ruby-qpid-0.7.946106-2.el5 > ruby-saslwrapper-0.1.934605-2.el5 > saslwrapper-0.1.934605-2.el5 > saslwrapper-devel-0.1.934605-2.el5 > sesame-0.10-1.el5 How reproducible: N/A Steps to Reproduce: 1. run 2. 3. Actual results: qpidd aborts during shutdown. Expected results: qpidd should not abort during shutdown. Additional info: [22:23:30] run(A) 61/100--------------------------[err_cnt:0] [22:23:40] .starting qpidd - 1] [22:23:40] .waiting for qpidd start - 2] . [22:23:43] ..qpidd just started [22:23:45] ..qpidd started on port 5672, parent lock file exists [22:23:45] ..qpidd process detected (11485) [22:23:45] .running txtest - 3] [22:23:45] ..config:[msg_per_tx:185, tx_cnt:10518, msg_cnt:3279, msg_size:223, q_cnt:13, --dtx no] .. [22:23:49] ..txtest just started (and transfering) [22:23:49] ..qpidd & txtest up and running [22:23:49] .wait some rand time & kill the qpidd (now waiting 8sec) - 4] [22:23:57] INFO:mrg_kill_process_id:processe[s] 11485 did not stop yet! [22:24:07] .pack the journal (before qpidd tries to recover) - 5] [22:24:37] ..qpidd journal saved in qpidd_journal_a0061.tar.bz2 [22:24:37] .dummy step (journal lock removal disabled) - 6] [22:24:42] ..qpidd & txtest and not running at the moment [22:24:42] .re-start qpidd for journal recovery - 7] [22:24:42] .wait for qpidd journal recovery settles - 8] . [22:24:45] ..qpidd just started [22:24:45] ..qpidd started on port 5672, lock is present [22:24:45] .re-running txtest - 9] [22:24:51] ..txtest went OK (ecode:0;nr.lines:16 txtest.transcript.log) [22:24:56] .stop qpidd/txtest (if needed) - 10] [22:24:56] ..killing /usr/sbin/qpidd (PID:12146) ./qpidd_runner.sh: line 27: 12146 Killed ${QPIDD_DIR}/qpidd ${RUNNER_QPIDD_PARAMS} >> ${QPIDD_TRANSCRIPT} 2>&1 [22:24:56] mrg_kill_process_id:processes 12146 stopped by signal[s] 9 [22:25:07] run(A) 62/100--------------------------[err_cnt:0] MARK-LWD-LOOP -- 2011-03-27 22:25:11 -- [22:25:17] .starting qpidd - 1] [22:25:17] .waiting for qpidd start - 2] . [22:25:19] ..qpidd just started [22:25:21] ..qpidd started on port 5672, parent lock file exists [22:25:21] ..qpidd process detected (12568) [22:25:21] .running txtest - 3] [22:25:21] ..config:[msg_per_tx:237, tx_cnt:12182, msg_cnt:3215, msg_size:261, q_cnt:6, --dtx no] . [22:25:24] ..txtest just started (and transfering) [22:25:24] ..qpidd & txtest up and running [22:25:24] .wait some rand time & kill the qpidd (now waiting 6sec) - 4] [22:25:30] INFO:mrg_kill_process_id:processe[s] 12568 did not stop yet! ./qpidd_runner.sh: line 27: 12568 Aborted (core dumped) ${QPIDD_DIR}/qpidd ${RUNNER_QPIDD_PARAMS} >> ${QPIDD_TRANSCRIPT} 2>&1 [22:25:40] .pack the journal (before qpidd tries to recover) - 5] [22:26:02] ..qpidd journal saved in qpidd_journal_a0062.tar.bz2 [22:26:02] .dummy step (journal lock removal disabled) - 6] [22:26:07] ..qpidd & txtest and not running at the moment [22:26:07] .re-start qpidd for journal recovery - 7] [22:26:07] .wait for qpidd journal recovery settles - 8] . [22:26:10] ..qpidd just started [22:26:10] ..qpidd started on port 5672, lock is present [22:26:10] .re-running txtest - 9] [22:26:16] ..txtest went OK (ecode:0;nr.lines:9 txtest.transcript.log) [22:26:21] .stop qpidd/txtest (if needed) - 10] [22:26:21] ..killing /usr/sbin/qpidd (PID:13123) ./qpidd_runner.sh: line 27: 13123 Killed ${QPIDD_DIR}/qpidd ${RUNNER_QPIDD_PARAMS} >> ${QPIDD_TRANSCRIPT} 2>&1 [22:26:21] mrg_kill_process_id:processes 13123 stopped by signal[s] 9 [22:26:32] run(A) 63/100--------------------------[err_cnt:0] (*): Shared library is missing debugging information. (gdb) 8 Thread 12568 0x0038e410 in __kernel_vsyscall () 7 Thread 12570 0x0038e410 in __kernel_vsyscall () 6 Thread 12576 0x002064ca in ?? () from /usr/lib/libaio.so.1 5 Thread 12577 0x002064ca in ?? () from /usr/lib/libaio.so.1 4 Thread 12582 0x0038e410 in __kernel_vsyscall () 3 Thread 12583 0x0038e410 in __kernel_vsyscall () 2 Thread 12589 0x0038e410 in __kernel_vsyscall () * 1 Thread 12569 0x0038e410 in __kernel_vsyscall () Thread 1 (Thread 12569): #0 0x0038e410 in __kernel_vsyscall () #1 0x009dedf0 in raise () from /lib/libc.so.6 #2 0x009e0701 in abort () from /lib/libc.so.6 #3 0x0051bb50 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/libstdc++.so.6 #4 0x00519555 in ?? () from /usr/lib/libstdc++.so.6 #5 0x00519592 in std::terminate() () from /usr/lib/libstdc++.so.6 #6 0x005196ca in __cxa_throw () from /usr/lib/libstdc++.so.6 #7 0x001c4e2c in mrg::journal::wmgr::get_events (this=0x9dc460c, state=UNUSED, timeout=0x0, flush=false) at jrnl/wmgr.cpp:676 #8 0x001929a9 in mrg::journal::jcntl::get_wr_events (this=0x9dc43cc, timeout=0x0) at jrnl/jcntl.cpp:363 #9 0x00140799 in mrg::msgstore::JournalImpl::getEventsFire (this=0x9dc43c8) at JournalImpl.cpp:505 #10 0x00140827 in mrg::msgstore::GetEventsFireEvent::fire (this=0x9dc41b8) at JournalImpl.cpp:52 #11 0x0077e245 in qpid::sys::TimerTask::fireTask (this=0x9dc41b8) at qpid/sys/Timer.cpp:57 #12 0x0077e2af in qpid::sys::Timer::fire (this=0x9d39048, t=...) at qpid/sys/Timer.cpp:188 #13 0x0077f693 in qpid::sys::Timer::run (this=0x9d39048) at qpid/sys/Timer.cpp:123 #14 0x00684341 in qpid::sys::(anonymous namespace)::runRunnable (p=0x9d39048) at qpid/sys/posix/Thread.cpp:35 #15 0x00b48832 in start_thread () from /lib/libpthread.so.0 #16 0x00a880ae in clone () from /lib/libc.so.6 (gdb) quit