Bug 758209

Summary: A rare condition can lead to fail to recv
Product: Red Hat Enterprise Linux 6 Reporter: Steven Dake <sdake>
Component: corosyncAssignee: Jan Friesse <jfriesse>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.2CC: cluster-maint, jkortus, jwest
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: corosync-1.4.1-5.el6 Doc Type: Bug Fix
Doc Text:
Previously, the range condition for the update_aru() function could cause incorrect check of message IDs. Due to this, in rare cases, the corosync utility entered the "FAILED TO RECEIVE" state, and so failed to receive multicast packets. With this update, the range value in the update_aru() function is no longer checked for; the fail_to_recv_const constant performs such checks. Now, corosync does not fail to receive packets.
Story Points: ---
Clone Of:
: 806901 (view as bug list) Environment:
Last Closed: 2012-06-20 12:23:03 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 791234, 791235, 791236, 806901    
Attachments:
Description Flags
Proposed patch none

Description Steven Dake 2011-11-29 14:49:01 UTC
Description of problem:
from the mailing list:
Hi Steven Dake:

Today, I have observed one of the reason that corosync running into
FAILED TO RECEIVE state.

There was five nodes(A,B,C,D,E) in my testing, and I limited the UDP
transmission rate of C nodes by iptables command:
iptables -A INPUT -i eth0 -p udp -m limit --limit 10000/s
--limit-burst 1 -j ACCEPT
iptables -A INPUT -i eth0 -p udp -j DROP


After one hour later, C node had been missing some MCAST messages,
it's state described as following:
==state of C node==
my_aru:0x805
my_high_seq_received:0xC2C
my_aru_count:7

=>receved MCAST message with seq:806 from B nodes
=>enter *message_handler_mcast*
  =>add this message to regular_sort_queue
  ...
  =>enter *update_aru* function
    => range = (my_high_seq_received - my_aru)
             = (0xC2C - 0x805)
             = 1063
    => if range>1024, do nothing and and return directly.
==END==

According this logic, after (my_high_req_received-my_aru)>1024, my_aru
will not be updated though corosync can receive MCAST messages
retransmitted by other nodes.

But at that timte, my_aru_count was only 7. So the corosync at C node
would keep in this status until my_aru_count increased to
fail_to_recv_const(the default value is 2500). This was a long time
for corosync, but we wasted it.

To solve this issue, maybe we can enlarge the range condition in
update_aru function? Or we just ingnore the checking of range value,
it seems no harmfull, because we have been using fail_to_recv_const to
control the things.

-- Yunkai Zhang Work at Taobao 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.there are steps above
2.
3.
  
Actual results:
failed to recv occurs at low seq ids

Expected results:
no failed to recv

Additional info:

Comment 2 Jan Friesse 2012-01-31 08:39:53 UTC
Created attachment 558565 [details]
Proposed patch

Comment 8 Jan Friesse 2012-03-07 08:53:18 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
Corosync contains improper check of messages ids

Consequence
Corosync can fail with message "FAILED TO RECEIVE" in some rare scenarios

Fix
Remove improper check

Result
Corosync no longer fails in specified scenarion with "FAILED TO RECEIVE"

Comment 9 Eliska Slobodova 2012-03-08 13:04:17 UTC
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,11 +1 @@
-Cause
+Previously, the range condition for the update_aru() function could cause incorrect check of message IDs. Due to this, in rare cases, the corosync utility entered the "FAILED TO RECEIVE" state, and so failed to receive multicast packets. With this update, the range value in the update_aru() function is no longer checked for; the fail_to_recv_const constant performs such checks. Now, corosync does not fail to receive packets.-Corosync contains improper check of messages ids
-
-Consequence
-Corosync can fail with message "FAILED TO RECEIVE" in some rare scenarios
-
-Fix
-Remove improper check
-
-Result
-Corosync no longer fails in specified scenarion with "FAILED TO RECEIVE"

Comment 12 errata-xmlrpc 2012-06-20 12:23:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0777.html