Bug 508124 - long scheduling delays of corosync process cause totem to meltdown
long scheduling delays of corosync process cause totem to meltdown
Status: CLOSED UPSTREAM
Product: Fedora
Classification: Fedora
Component: corosync (Show other bugs)
rawhide
All Linux
low Severity medium
: ---
: ---
Assigned To: Steven Dake
Fedora Extras Quality Assurance
:
Depends On:
Blocks: 508128
  Show dependency treegraph
 
Reported: 2009-06-25 13:46 EDT by Steven Dake
Modified: 2016-04-26 10:17 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 508128 (view as bug list)
Environment:
Last Closed: 2009-07-06 22:44:34 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Steven Dake 2009-06-25 13:46:59 EDT
Description of problem:
When a process pauses for longer then the token timeout, the other
processors in the system form a new ring.  The remaining processor then
eventually reschedules and processes the pending membership multicast
messages in its kernel queues.  This wreaks havok on the membership of
the other nodes.

While a proper kernel shouldn't pause for long periods, its a reality
that many kernels still have long periods of spinlocking without
scheduling and no proper preemption.

This patch resolves the scenario by creating a timer which records a
time stamp at an interval that is the token timeout / 5.  Then if a
process executes the membership algorithm by receiving a join message,
the current time is retrieved and compared to the timestamp.  If they
differ by more then token timeout / 2, it is assumed the process
couldn't schedule (because it couldn't trigger the timer callbacks via
poll) and calls totemnet to flush any pending multicasts in the file
descriptor responsible for receiving multicast messages.  This results
in the old membership messages being thrown away allowing the new
membership to form properly.

This can be tested by ctrl-z a corosync process in a 8 node cluster.
Then use fg to bring it into the foreground.  Pre-patch - bad news -
post patch, prints a notice and proceeds properly.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.setup 8 node cluster
2.ctrl-z 1 node
3.wait until other nodes form new ring
4. fg ctrl-z node
  
Actual results:
totem membership explodes

Expected results:
new ring formed properly

Additional info:
Comment 1 Steven Dake 2009-06-25 13:52:09 EDT
patch posted to ml.

Note You need to log in before you can comment on or make changes to this bug.