Bug 922671
| Summary: | CPG: Corosync can duplicate and/or lost messages - Local IPC | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Jan Friesse <jfriesse> |
| Component: | corosync | Assignee: | Jan Friesse <jfriesse> |
| Status: | CLOSED ERRATA | QA Contact: | Cluster QE <mspqa-list> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 6.4 | CC: | bmarson, cluster-maint, jkortus, perfbz, sbradley, sdake |
| Target Milestone: | rc | Keywords: | ZStream |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | corosync-1.4.1-16.el6 | Doc Type: | Bug Fix |
| Doc Text: |
When running applications which used the Corosync IPC library, some messages in the dispatch() function were lost or duplicated. This update properly checks the return values of the dispatch_put() function, returns the correct remaining bytes in the IPC ring buffer, and ensures that the IPC client is correctly informed about the real number of messages in the ring buffer. Now, messages in the dispatch() function are no longer lost or duplicated.
|
Story Points: | --- |
| Clone Of: | 907894 | Environment: | |
| Last Closed: | 2013-11-21 04:33:08 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 907894, 929096, 929098, 929100, 929101 | ||
| Attachments: | |||
|
Comment 1
Jan Friesse
2013-03-18 09:26:32 UTC
Created attachment 711845 [details]
Proposed patch - part 1
Created attachment 711846 [details] Proposed patch - part 2 - check dispatch_put return code Proposed patch 1 + 2 are reproducible by running https://github.com/jfriesse/csts/blob/master/tests/start-cfgstop-one-by-one-with-load.sh. When bug appear, there are duplicated messages in output (usually last 2 are duplicate). Created attachment 711848 [details] Proposed patch - part 3 - Take alignment in acount for free_bytes in ring buffer "Unit test" https://github.com/jfriesse/csts/blob/master/tests/ipc-overflow.sh Created attachment 711850 [details]
Proposed patch - part 4 - Properly lock pending_semops
Sadly, this problem is race so it's quite hard to reproduce. I had moderate success with two nodes and:
- node 1 - running corosync, cpgload -q -n 500 and cpgload -l 1 -n 500 -q
- node 2 - running corosync and cpgload -q -n 500
After 5+ hours, one of cpgload is terminated (it ends up return code 0, because CS_ERR_LIBRARY arrived).
With patch, I was able to run configuration above for 3 days.
Keep in mind that it CAN happen, that (because of extreme high load) cpgload may pause and corosync is terminated by OOM. This is not a bug.
Barry: Can you please give a try scratch build http://brewweb.devel.redhat.com/brew/taskinfo?taskID=5527619 ? I was able to run above test for 3 days, and I really hope it solves problem you are hitting (please use most unstable configuration, so irqbalance, corosync unpined, errata kernel, ...) Created attachment 711875 [details]
Proposed patch - part 4 - Properly lock pending_semops - Try2
Created attachment 711877 [details]
Proposed patch - part 4 - Properly lock pending_semops - Try3
I have run SAS calibration on my 4 node cluster with the latest bits. 5 different types of test configurations were run to try and stress the cluster interconnect in different fashions Each configuration was run 10 times (each of which stresses the system for at least 2-3 hours). After 6 days of testing, I can report ZERO failures. Nice job. Barry Verified using ipc-overflow.sh test: FAIL on corosync-1.4.1-15.el6.x86_64 (RHEL6.4) PASS on corosync-1.4.1-17.el6.x86_64 (RHEL6.5) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-1531.html |