Corosync running on multiple nodes, some of nodes are killed (corosync dies/exits/switch failure/...)
Very rarely, corosync can lost or duplicate messages.
Fixed many race conditions.
Corosync should no longer loose or duplicate messages.
Description of problem:
Corosync can duplicate messages on corosync server exit and/or lost (overwrite) messages (looks like generally anytime)
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. https://github.com/jfriesse/csts/blob/master/tests/start-cfgstop-one-by-one-with-load.sh (this is "Unit" test)
- On the corosync exit, multiple duplicated messages can be delivered to app
- Messages are lost/overwritten
Test result is 0 ($? == 0)
Created attachment 693380 [details]
Proposed patch - part 1
Created attachment 693381 [details]
Proposed patch - part 2 - check dispatch_put return code
With patches 1 and 2, duplicate messages are gone. Problem with corrupted/lost messages persists.
Created attachment 706733 [details]
Proposed patch - part 3 - Take alignment in acount for free_bytes in ring buffer
This patch solves another issue (CRITICAL). I believe it also solves problem's bmarsons is hitting.
- Start corosync on one node
- Run cpgload -q -n 500
- After (usually) less then three minutes, lost messages will appear in form:
20130307T170546:(a126220a 3200):341:Incorrect msg seq 341 != 297
20130307T170546:(a126220a 3200):342:Incorrect msg seq 342 != 297
- cpgload will end with
Dispatch error 2
"Unit test" for proposed patch - part 3:
With patch 3, messages are no longer lost on single node cluster, but sadly still may be lost on multiple nodes cluster.
First run (10 iterations) showed no failures :) .. Running again for more confidence.
I've split this BZ to two different BZillas.
Bug #907894 (this one) is to solve multiple nodes message corruption/lost/out of order messages delivery.
Bug #922671 is to solve local IPC problems.
Tho marking patches in this BZ as obsoleted (and move them to Bug #922671)
Created attachment 713841 [details]
Proposed patch - part 1 - totempg: Make iov_delv local variable
Created attachment 713842 [details]
Proposed patch - part 2 - Remove exit thread and replace it by exit pipe
Created attachment 713843 [details]
Proposed patch - part 3 - schedwrk: Set values before create callback
Created attachment 713844 [details]
Proposed patch - part 4 - Fix race for sending_allowed
Created attachment 713855 [details]
Proposed patch - part 5 - totempg: Store and restore global variables
Created attachment 754386 [details]
Proposed patch - part 6 - Lock sync_in_process variable
sync_in_process is changed by coropoll thread (main thread) but used by
all IPC connections. To ensure correct value is read, mutex is added.
Verified using start-cfgstop-one-by-one-with-load.sh test
FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4)
PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5)
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.