Bug 525813

Fixed in r.819600.

When running tsxtest to test flow-to-disk, the broker will occasionally close the client session with RHM_IORES_EMPTY - which indicates that the read pipeline has nothing more to read when flow-to-disk content is reloaded from the store.

On a 8- or 16-core box, start a broker:
./qpidd --tcp-nodelay --worker-threads 4 --auth no --load-module ../../../store/lib/.libs/msgstore.so --data-dir /tmp --log-enable info+

In another window, start the client:
./tsxtest --flow 1 --cut_depth 1 --durable_msg yes --rate 1000 --messages 40000

If the fault occurs, the RHM_IORES_EMPTY message can be seen in the broker window; the client has either closed or crashed.

This fault has been difficult to reproduce, with one or two occurrences per day.

The problem is out-of-order enqueueing. While the rids are assigned in order, the operation of assigning and enqueueing is not atomic. When multiple threads are enqueueing, occasionally a thread with an assigned rid will be stopped and another thread with a later assigned rid will enqueue its record first.

The logic used in the JournalImpl::loadContent() method does not take this possibility into account, and will run to the end of all the known records and return RHM_IORES_EMPTY if it encounters an out-of-order rid on disk.

This is now fixed by keeping a list of out-of-order rids that are greater than the current rid being read which may be encountered while reading. If the following read has its rid in this list, then the read pipeline is invalidated.

Fixed in store svn r.3650; this is synced with qpid r.819600.

git commit on branch mrg_1.1.x: http://git.et.redhat.com/git/qpid.git/?p=qpid.git;a=commitdiff;h=f6518f49336ff36826e3780fa72973e5da0b6891

Included in store backport for 1.2.

Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Resolved problem with out-of-order enqueueing causing the broker to terminate the client session incorrectly and the client to close or crash
(525813)

The issue has been fixed, no occurence found after extensive stress testing
using above scenario on RHEL 4.8 / 5.4 i386 / x86_64 on packages:
[root@mrg-qe-02 ~]# rpm -qa | egrep '(rhm|qpid)' | sort -u
python-qpid-0.5.752581-4.el5
qpidc-0.5.752581-29.el5
qpidc-debuginfo-0.5.752581-29.el5
qpidc-devel-0.5.752581-29.el5
qpidc-perftest-0.5.752581-29.el5
qpidc-rdma-0.5.752581-29.el5
qpidc-ssl-0.5.752581-29.el5
qpidd-0.5.752581-29.el5
qpidd-acl-0.5.752581-29.el5
qpidd-cluster-0.5.752581-29.el5
qpidd-devel-0.5.752581-29.el5
qpid-dotnet-0.4.738274-2.el5
qpidd-rdma-0.5.752581-29.el5
qpidd-ssl-0.5.752581-29.el5
qpidd-xml-0.5.752581-29.el5
qpid-java-client-0.5.751061-9.el5
qpid-java-common-0.5.751061-9.el5
rhm-0.5.3206-16.el5
rhm-debuginfo-0.5.3206-16.el5
rhm-docs-0.5.756148-1.el5
rh-qpid-tests-0.5.752581-29.el5

-> VERIFIED

Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,2 +1,9 @@
-Resolved problem with out-of-order enqueueing causing the broker to terminate the client session incorrectly and the client to close or crash
+Messaging bug fix
-(525813)+
+C: When multiple threads are enqueueing, occasionally a thread with an assigned rid will be stopped and another thread with a later assigned rid will enqueue its record first.
+C: The broker will close the client session with an error.
+F: A list is now kept of out-of-order rids that are greater than the current rid being read and which could be encountered while reading. If the
+following read has its rid in this list, then the read pipeline is invalidated.
+R: The crash no longer occurs.
+
+When multiple threads were enqueueing, occasionally a thread with an assigned rid would stop and another thread with a later assigned rid would enqueue its record first. This caused the  broker to close the client session with an error. A list is now kept of out-of-order rids that are greater than the current rid being read and which could be encountered. If the next read has a rid in the list, then the read pipeline is invalidated. This prevents the crash from occurring.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-1633.html