Red Hat Bugzilla – Bug 907894
CPG: Corosync can duplicate and/or lost messages - Multiple nodes problems
Last modified: 2015-09-27 22:24:16 EDT
Description of problem: Corosync can duplicate messages on corosync server exit and/or lost (overwrite) messages (looks like generally anytime) Version-Release number of selected component (if applicable): EL 6.4 How reproducible: Almost 100% Steps to Reproduce: 1. https://github.com/jfriesse/csts/blob/master/tests/start-cfgstop-one-by-one-with-load.sh (this is "Unit" test) Actual results: - On the corosync exit, multiple duplicated messages can be delivered to app - Messages are lost/overwritten Expected results: Test result is 0 ($? == 0) Additional info:
Created attachment 693380 [details] Proposed patch - part 1
Created attachment 693381 [details] Proposed patch - part 2 - check dispatch_put return code
With patches 1 and 2, duplicate messages are gone. Problem with corrupted/lost messages persists.
Created attachment 706733 [details] Proposed patch - part 3 - Take alignment in acount for free_bytes in ring buffer This patch solves another issue (CRITICAL). I believe it also solves problem's bmarsons is hitting. Reproducer: - Start corosync on one node - Run cpgload -q -n 500 - After (usually) less then three minutes, lost messages will appear in form: 20130307T170546:(a126220a 3200):341:Incorrect msg seq 341 != 297 20130307T170546:(a126220a 3200):342:Incorrect msg seq 342 != 297 - cpgload will end with Dispatch error 2
"Unit test" for proposed patch - part 3: https://github.com/jfriesse/csts/commit/5188f85d1956db1c14e37737d4dadd5935c78d52
With patch 3, messages are no longer lost on single node cluster, but sadly still may be lost on multiple nodes cluster.
First run (10 iterations) showed no failures :) .. Running again for more confidence. Barry
I've split this BZ to two different BZillas. Bug #907894 (this one) is to solve multiple nodes message corruption/lost/out of order messages delivery. Bug #922671 is to solve local IPC problems. Tho marking patches in this BZ as obsoleted (and move them to Bug #922671)
Created attachment 713841 [details] Proposed patch - part 1 - totempg: Make iov_delv local variable
Created attachment 713842 [details] Proposed patch - part 2 - Remove exit thread and replace it by exit pipe
Created attachment 713843 [details] Proposed patch - part 3 - schedwrk: Set values before create callback
Created attachment 713844 [details] Proposed patch - part 4 - Fix race for sending_allowed
Created attachment 713855 [details] Proposed patch - part 5 - totempg: Store and restore global variables
Created attachment 754386 [details] Proposed patch - part 6 - Lock sync_in_process variable sync_in_process is changed by coropoll thread (main thread) but used by all IPC connections. To ensure correct value is read, mutex is added.
Verified using start-cfgstop-one-by-one-with-load.sh test FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4) PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1531.html