This service will be undergoing maintenance at 00:00 UTC, 2016-09-28. It is expected to last about 1 hours
Bug 140414 - Piranha/LVS of RH Cluster Suite doesn't re-arp vip after a split-brain scenario
Piranha/LVS of RH Cluster Suite doesn't re-arp vip after a split-brain scenario
Status: CLOSED ERRATA
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: piranha (Show other bugs)
3
All Linux
medium Severity medium
: ---
: ---
Assigned To: Lon Hohberger
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2004-11-22 15:27 EST by Lon Hohberger
Modified: 2009-04-16 16:15 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-11-23 09:56:13 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Patch fixing gratuitous ARP problem after scenario (9.05 KB, patch)
2004-11-22 15:29 EST, Lon Hohberger
no flags Details | Diff
Patch fixing race condition (4.55 KB, patch)
2004-11-22 15:43 EST, Lon Hohberger
no flags Details | Diff

  None (edit)
Description Lon Hohberger 2004-11-22 15:27:46 EST
Description of problem:

When Piranha detects a "split-brain" scenario it will not re-arp the
VIP from the primary LVS director node.  This is observed when running
LVS in direct routing mode.  It causes the communication to virtual IP
address to go to the wrong node until the upstream router's ARP cache
expires, which can be quite a long time (4 hours in some cases).
Comment 1 Lon Hohberger 2004-11-22 15:29:48 EST
Created attachment 107236 [details]
Patch fixing gratuitous ARP problem after scenario

This patch fixes piranha so that gratuitous ARPs are sent after a split brain.
Comment 2 Lon Hohberger 2004-11-22 15:42:17 EST
There is also a race condition, even with the above patch applied.  It
can be visualized in the following manner:

Working case:

Master	       Backup
active ------>
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
arp... <------ active
active ------> (shuts down)
       <------ inact

Non-working case: the master doesn't send a new gratuitous ARP after
the split-brain:

Master	       Backup
active ------>
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
	  Z <- active
active ------> (shuts down)
       <------ inact


So, we need to ensure that the backup always notifies the master that
it is or was the master so that the master sends gratuitous arps for
the virtual IP address.  This can be done by using a "dying breath"
heartbeat packet.  When backup receives a heartbeat from the master,
it can send one last ACTIVE packet prior to shutting down LVS
services.  This can cause the master to send gratuitous arps twice in
some cases, but this should not cause problems:

Master	       Backup
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
arp... <------ active
active ------>                        }
       <------ dying "ACTIVE" packet  } arp sent a second time
arp...         (shuts down)           }
       <------ inact
active ------>
Comment 3 Lon Hohberger 2004-11-22 15:43:49 EST
Created attachment 107238 [details]
Patch fixing race condition
Comment 4 Lon Hohberger 2004-11-22 15:45:52 EST
The patch ensures that in the non-working case that at least one
gratuitous ARP is sent for the virtual IP address; the fact that two
are sent in the working case is a side effect.

There are other ways to go about this; however, this is the simplest.
Comment 6 Lon Hohberger 2004-11-22 16:26:50 EST
One way to reproduce the original split-brain is to simply 'killall
-STOP pulse' on the master while monitoring /var/log/messages on the
backup.

When the backup takes over and sends ARPs, type 'killall -CONT pulse'
on the master.  Sometimes it will send arps; other times it won't.
Comment 7 Lon Hohberger 2004-11-22 16:28:34 EST
Correction: The previous comment is how to recreate the race condition
with relative ease.
Comment 8 Lon Hohberger 2004-11-23 09:56:13 EST
This will go out as an errata in the next update.

Note You need to log in before you can comment on or make changes to this bug.