Red Hat Bugzilla – Bug 614219
token timer is reset on each received retransmitted token resulting in membership meltdown in some conditions
Last modified: 2016-04-26 11:02:22 EDT
Description of problem:
The totem specification is clear:
When a retransmitted token is received, it should be dropped
When a new token is received, it should reset the token timeout
This enables the timers related to the token expiration to happen at about the same time. In the case where the timer is reset on each token retransmission, it is possible for some nodes to be in operational (because they keep reseting the token loss timeout) while other nodes have detected a failure. A token loss should be detected by all nodes when not having received the token. A retransmitted token keeps resetting the token timeout.
To see this in practice, consider a 4 node cluster with token=5000 (5sec) and retransmit rate of 1.5 seconds. One of the nodes will still be in operational because there is a cascade of token loss events that occur from n1 (waits 5 seconds), (waits 5 seconds) to n2, to (waits 5 seconds) n3 intervals. When reaching the 3rd node, the 3rd node thinks everything is fine when in fact it has failed to receive its token within its allotted timeout, violating the proof of the algorithm...
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.start 4 node corosync cluster
2.set token=5000, consensus=7000, join=60
3.ctrl-z one of the four nodes (ctrl-c is different, it sends a special message to exit the node)
the membership protocol melts down and bad things happen (tm)
token loss is detected by all nodes reasonably.
1 liner patch
Created attachment 431603 [details]
patch to fix problem
Verified w/ corosync-1.2.3-17.el6 using the steps to reproduce.
Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
An internal timer variable was reset on each token retransmission rather than only on original token transmission; this has been fixed in this updated package.