Description of problem: If a standalone, non-clustered qpidd is killed while in the middle of servicing a client, it subsequently fails to set the redelivered status for those messages that were already delivered to the client but not acknowledged. Version-Release number of selected component (if applicable): 1.1.1 How reproducible: 100% Steps to Reproduce: (general -- see below for more detail) 1. put durable messages on a durable queue 2. while reading the messages with a client app, stop the broker ('/etc/init.d/qpid stop') 3. start the broker, and print out the delivery_properties of the first message received after the broker starts up again Actual results: 'redelivered' is not set as one of the delivery_properties Expected results: 'redelivered=True' is one of the delivery_properties Additional info: Sample python client code included. 1. gunzip into a directory. Create two terminal windows. Use one to start and stop the broker and use the other to run the python clients. 2. With the broker running, run 'python newq.py' to create the queue. 3. Run 'python putmsg.py' to load the messages into the queue. 3. Run 'python getmsg.py' for a few seconds then Ctrl-C. 4. Run 'python getmsg.py' again to see that the first message has the redelivered flag in the deliver_properties. Let getmsg finish. 5. Rerun 'python putmsg.py' to reload the messages. 6. Run 'python getmsg.py'. 7. After a few seconds, stop the broker 'sudo /etc/init.d/qpidd stop'. 8. Restart the broker 'sudo /etc/init.d/qpidd start'. 9. Note that the deliver_properties of the first message after the restart do not include the redelivered flag even though you can tell by the message numbering that it's been received before.
Created attachment 349449 [details] Test code to reproduce the problem
A simple solution is to mark all recovered messages as potentially redelivered. That avoids any false negatives but obviously may result in a lot of false positives. To improve on this, the intention to deliver a particular message (or up to a particular mesage) would need to be recorded in the journal (and flushed to disk) before the delivery was attempted. Avoiding hefty performance penalties for this would be important (and/or making it an optional feature).
Checked in a change which sets the Message::redelivered flag true on recovery of all messages. qpid r.891362 store r.3747 There are no tests for this yet, waiting for an update to the python API which will allow examination of the redelivered flag on a retrieved message (see bug 548141).
Test for redelivered flag being set on recovery added in r.3751.
Customer reports (IT 384493) that JMSRedelivered flag is not set during recovery Issue is noted with 1. Standalone broker [at restart] 2. Clustered broker [at failover] When this is tested, Java client should be verified as well.
The python part (both cases, interruption of the client|restart of the broker) of the issue was verified and is functional. Final feedback on java part soon.
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Cause: Recovery of Store (by restarting broker with durable messages) never sets the redelivered flag when sending the recovered messages which have been previously sent but not acknowledged. Consequence: The Client is not notified that this message has been previously sent, and could potentially have already processed it, possibly resulting in a duplicate. Fix: Because the store does not save the send state, it now marks _all_ recovered messages as redelivered. Results: All recovered messages are marked as redelivered. This is in effect switching false positives for false negatives, which is better from a reliability point of veiw.
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1,4 +1 @@ -Cause: Recovery of Store (by restarting broker with durable messages) never sets the redelivered flag when sending the recovered messages which have been previously sent but not acknowledged. +After a broker restart, when a durable message was recovered and sent again, it was not marked as "redelivered" if it had been previously sent but not acknowledged. As a result, the client was not notified of this fact, and the message may have been duplicated. With this update, all recovered messages are now marked as "redelivered", making the message delivery more reliable.-Consequence: The Client is not notified that this message has been previously sent, and could potentially have already processed it, possibly resulting in a duplicate. -Fix: Because the store does not save the send state, it now marks _all_ recovered messages as redelivered. -Results: All recovered messages are marked as redelivered. This is in effect switching false positives for false negatives, which is better from a reliability point of veiw.
The testing status: The python case: a] re-running client after client stopped by signal All messages are marked 'Redelivered' when client is restarted - OK b] rerunning client after client disconnected at broker's shutdown All messages are marked 'Redelivered' when client is restarted - OK The java case: a] re-running client after client stopped by signal There are just few tens of messages marked 'Redelivered' after client is restarted - This behavior is in contrary of comment's 2 definition. b] rerunning client after client disconnected at broker's shutdown All messages are marked 'Redelivered' when client is restarted - OK c] client run after failover event There are just few tens of messages marked 'Redelivered' after client fails over to another clustered broker - This behavior is in contrary of comment's 2 definition. Raising needinfo for crosscheck and keeping the state.
Created attachment 451839 [details] The log from java failover message redelivered testing
The action of setting the redelivered flag on *all* recovered messages applies to the shutdown and restart of the broker only. If a client is shut down, then only the unacknowledged messages being resent will have the redelivered flags set, as expected. Python: a) Not correct. The broker has not been restarted, and thus only the messages which were actually sent but not acknowledged should have the redelivered flag set. This case should behave the same as Java case a). Make sure acknowledgements are enabled. If this is indeed the case with acks enabled, then raise a separate bug. You may want to check with Rafi on this. b) Correct, since the broker has been restarted and the messages in the store recovered. Java: a) Correct, since the broker has not been restarted and there has been no store recovery. Only messages actually sent but not acknowledged will be marked redelivered. b) Correct, since the broker has been restarted. c) I am uncertain of how clustering behaves in this case, but it sounds correct. Similar to case a), the broker has not been restarted. Check with Alan.
The behavior was discussed with following resolution: Python case: a] incorrect, addressed as separate issue: bug 640618 (more specific problem which can be excluded from this defect definition - confirmed) b] confirmed as correct Java case: a] confirmed as correct b] confirmed as correct c] confirmed as correct Conclusion: Qpidd broker sets correctly message's redelivered flag for case when broker is restarted and also when failover happened. The issue has been fixed, tested on RHEL 4.8 / 5.5 i386 / x86_64 on packages: python-qmf-0.7.946106-13.el5 python-qpid-0.7.946106-14.el5 qmf-0.7.946106-17.el5 qmf-devel-0.7.946106-17.el5 qpid-cpp-client-0.7.946106-17.el5 qpid-cpp-client-devel-0.7.946106-17.el5 qpid-cpp-client-devel-docs-0.7.946106-17.el5 qpid-cpp-client-rdma-0.7.946106-17.el5 qpid-cpp-client-ssl-0.7.946106-17.el5 qpid-cpp-mrg-debuginfo-0.7.946106-17.el5 qpid-cpp-server-0.7.946106-17.el5 qpid-cpp-server-cluster-0.7.946106-17.el5 qpid-cpp-server-devel-0.7.946106-17.el5 qpid-cpp-server-rdma-0.7.946106-17.el5 qpid-cpp-server-ssl-0.7.946106-17.el5 qpid-cpp-server-store-0.7.946106-17.el5 qpid-cpp-server-xml-0.7.946106-17.el5 qpid-dotnet-0.4.738274-2.el5 qpid-java-client-0.7.946106-10.el5 qpid-java-common-0.7.946106-10.el5 qpid-tools-0.7.946106-11.el5 ruby-qmf-0.7.946106-17.el5 ruby-qpid-0.7.946106-2.el5 -> VERIFIED
Technical note updated. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. Diffed Contents: @@ -1 +1 @@ -After a broker restart, when a durable message was recovered and sent again, it was not marked as "redelivered" if it had been previously sent but not acknowledged. As a result, the client was not notified of this fact, and the message may have been duplicated. With this update, all recovered messages are now marked as "redelivered", making the message delivery more reliable.+After a broker restart, when a durable message was recovered and sent again, it was not marked as "redelivered" if it had been previously sent but not acknowledged. As a result, the client was not notified of this fact, and the message may have been duplicated. With this update, all recovered messages are now marked as "redelivered", thus making message delivery more reliable.
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2010-0773.html