Bug 907894 - CPG: Corosync can duplicate and/or lost messages - Multiple nodes problems
Summary: CPG: Corosync can duplicate and/or lost messages - Multiple nodes problems
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: corosync
Version: 6.4
Hardware: All
OS: All
urgent
urgent
Target Milestone: rc
: ---
Assignee: Jan Friesse
QA Contact: Cluster QE
URL:
Whiteboard:
Keywords:
Depends On: 922671 924261
Blocks: 960054
TreeView+ depends on / blocked
 
Reported: 2013-02-05 13:36 UTC by Jan Friesse
Modified: 2015-09-28 02:24 UTC (History)
7 users (show)

(edit)
Cause: 
Corosync running on multiple nodes, some of nodes are killed (corosync dies/exits/switch failure/...)

Consequence: 
Very rarely, corosync can lost or duplicate messages.

Fix: 
Fixed many race conditions.

Result: 
Corosync should no longer loose or duplicate messages.
Clone Of:
: 922671 (view as bug list)
(edit)
Last Closed: 2013-11-21 04:32:17 UTC


Attachments (Terms of Use)
Proposed patch - part 1 (1.56 KB, patch)
2013-02-05 13:38 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 2 - check dispatch_put return code (4.58 KB, patch)
2013-02-05 13:38 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 3 - Take alignment in acount for free_bytes in ring buffer (1.15 KB, patch)
2013-03-07 17:00 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 1 - totempg: Make iov_delv local variable (868 bytes, patch)
2013-03-21 13:23 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 2 - Remove exit thread and replace it by exit pipe (4.38 KB, patch)
2013-03-21 13:27 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 3 - schedwrk: Set values before create callback (1.11 KB, patch)
2013-03-21 13:28 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 4 - Fix race for sending_allowed (1.42 KB, patch)
2013-03-21 13:29 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 5 - totempg: Store and restore global variables (12.50 KB, patch)
2013-03-21 13:32 UTC, Jan Friesse
no flags Details | Diff
Proposed patch - part 6 - Lock sync_in_process variable (2.07 KB, patch)
2013-05-29 13:47 UTC, Jan Friesse
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1531 normal SHIPPED_LIVE corosync bug fix and enhancement update 2013-11-21 00:40:57 UTC

Description Jan Friesse 2013-02-05 13:36:33 UTC
Description of problem:
Corosync can duplicate messages on corosync server exit and/or lost (overwrite) messages (looks like generally anytime)

Version-Release number of selected component (if applicable):
EL 6.4

How reproducible:
Almost 100%

Steps to Reproduce:
1. https://github.com/jfriesse/csts/blob/master/tests/start-cfgstop-one-by-one-with-load.sh (this is "Unit" test)
  
Actual results:
- On the corosync exit, multiple duplicated messages can be delivered to app
- Messages are lost/overwritten

Expected results:
Test result is 0 ($? == 0)

Additional info:

Comment 1 Jan Friesse 2013-02-05 13:38:25 UTC
Created attachment 693380 [details]
Proposed patch - part 1

Comment 2 Jan Friesse 2013-02-05 13:38:57 UTC
Created attachment 693381 [details]
Proposed patch - part 2 - check dispatch_put return code

Comment 3 Jan Friesse 2013-02-05 13:39:36 UTC
With patches 1 and 2, duplicate messages are gone. Problem with corrupted/lost messages persists.

Comment 4 Jan Friesse 2013-03-07 17:00:24 UTC
Created attachment 706733 [details]
Proposed patch - part 3 - Take alignment in acount for free_bytes in ring buffer

This patch solves another issue (CRITICAL). I believe it also solves problem's bmarsons is hitting.

Reproducer:
- Start corosync on one node
- Run cpgload -q -n 500
- After (usually) less then three minutes, lost messages will appear in form:
20130307T170546:(a126220a 3200):341:Incorrect msg seq 341 != 297
20130307T170546:(a126220a 3200):342:Incorrect msg seq 342 != 297
- cpgload will end with
Dispatch error 2

Comment 6 Jan Friesse 2013-03-08 09:21:11 UTC
"Unit test" for proposed patch - part 3:
https://github.com/jfriesse/csts/commit/5188f85d1956db1c14e37737d4dadd5935c78d52

Comment 7 Jan Friesse 2013-03-08 10:00:41 UTC
With patch 3, messages are no longer lost on single node cluster, but sadly still may be lost on multiple nodes cluster.

Comment 10 Barry Marson 2013-03-10 15:08:42 UTC
First run (10 iterations) showed no failures :) .. Running again for more confidence.

Barry

Comment 11 Jan Friesse 2013-03-18 09:30:20 UTC
I've split this BZ to two different BZillas.
Bug #907894 (this one) is to solve multiple nodes message corruption/lost/out of order messages delivery.
Bug #922671 is to solve local IPC problems.

Tho marking patches in this BZ as obsoleted (and move them to Bug #922671)

Comment 12 Jan Friesse 2013-03-21 13:23:22 UTC
Created attachment 713841 [details]
Proposed patch - part 1 - totempg: Make iov_delv local variable

Comment 13 Jan Friesse 2013-03-21 13:27:13 UTC
Created attachment 713842 [details]
Proposed patch - part 2 - Remove exit thread and replace it by exit pipe

Comment 14 Jan Friesse 2013-03-21 13:28:25 UTC
Created attachment 713843 [details]
Proposed patch - part 3 - schedwrk: Set values before create callback

Comment 15 Jan Friesse 2013-03-21 13:29:47 UTC
Created attachment 713844 [details]
Proposed patch - part 4 - Fix race for sending_allowed

Comment 16 Jan Friesse 2013-03-21 13:32:17 UTC
Created attachment 713855 [details]
Proposed patch - part 5 - totempg: Store and restore global variables

Comment 18 Jan Friesse 2013-05-29 13:47:51 UTC
Created attachment 754386 [details]
Proposed patch - part 6 - Lock sync_in_process variable

sync_in_process is changed by coropoll thread (main thread) but used by
all IPC connections. To ensure correct value is read, mutex is added.

Comment 25 Jaroslav Kortus 2013-09-11 13:07:22 UTC
Verified using start-cfgstop-one-by-one-with-load.sh test

FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4)
PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5)

Comment 27 errata-xmlrpc 2013-11-21 04:32:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1531.html


Note You need to log in before you can comment on or make changes to this bug.