Bug 140414 - Piranha/LVS of RH Cluster Suite doesn't re-arp vip after a split-brain scenario
Summary: Piranha/LVS of RH Cluster Suite doesn't re-arp vip after a split-brain scenario
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Cluster Suite
Classification: Retired
Component: piranha (Show other bugs)
(Show other bugs)
Version: 3
Hardware: All Linux
medium
medium
Target Milestone: ---
Assignee: Lon Hohberger
QA Contact: Cluster QE
URL:
Whiteboard:
Keywords:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2004-11-22 20:27 UTC by Lon Hohberger
Modified: 2009-04-16 20:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-11-23 14:56:13 UTC
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Patch fixing gratuitous ARP problem after scenario (9.05 KB, patch)
2004-11-22 20:29 UTC, Lon Hohberger
no flags Details | Diff
Patch fixing race condition (4.55 KB, patch)
2004-11-22 20:43 UTC, Lon Hohberger
no flags Details | Diff

Description Lon Hohberger 2004-11-22 20:27:46 UTC
Description of problem:

When Piranha detects a "split-brain" scenario it will not re-arp the
VIP from the primary LVS director node.  This is observed when running
LVS in direct routing mode.  It causes the communication to virtual IP
address to go to the wrong node until the upstream router's ARP cache
expires, which can be quite a long time (4 hours in some cases).

Comment 1 Lon Hohberger 2004-11-22 20:29:48 UTC
Created attachment 107236 [details]
Patch fixing gratuitous ARP problem after scenario

This patch fixes piranha so that gratuitous ARPs are sent after a split brain.

Comment 2 Lon Hohberger 2004-11-22 20:42:17 UTC
There is also a race condition, even with the above patch applied.  It
can be visualized in the following manner:

Working case:

Master	       Backup
active ------>
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
arp... <------ active
active ------> (shuts down)
       <------ inact

Non-working case: the master doesn't send a new gratuitous ARP after
the split-brain:

Master	       Backup
active ------>
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
	  Z <- active
active ------> (shuts down)
       <------ inact


So, we need to ensure that the backup always notifies the master that
it is or was the master so that the master sends gratuitous arps for
the virtual IP address.  This can be done by using a "dying breath"
heartbeat packet.  When backup receives a heartbeat from the master,
it can send one last ACTIVE packet prior to shutting down LVS
services.  This can cause the master to send gratuitous arps twice in
some cases, but this should not cause problems:

Master	       Backup
       <------ inact
active ------>
       <------ inact
active -> Z
	  Z <- inact
active -> Z
	  Z <- inact
active -> Z    (activate / send arps)
	  Z <- active
active -> Z
arp... <------ active
active ------>                        }
       <------ dying "ACTIVE" packet  } arp sent a second time
arp...         (shuts down)           }
       <------ inact
active ------>


Comment 3 Lon Hohberger 2004-11-22 20:43:49 UTC
Created attachment 107238 [details]
Patch fixing race condition

Comment 4 Lon Hohberger 2004-11-22 20:45:52 UTC
The patch ensures that in the non-working case that at least one
gratuitous ARP is sent for the virtual IP address; the fact that two
are sent in the working case is a side effect.

There are other ways to go about this; however, this is the simplest.

Comment 6 Lon Hohberger 2004-11-22 21:26:50 UTC
One way to reproduce the original split-brain is to simply 'killall
-STOP pulse' on the master while monitoring /var/log/messages on the
backup.

When the backup takes over and sends ARPs, type 'killall -CONT pulse'
on the master.  Sometimes it will send arps; other times it won't.

Comment 7 Lon Hohberger 2004-11-22 21:28:34 UTC
Correction: The previous comment is how to recreate the race condition
with relative ease.

Comment 8 Lon Hohberger 2004-11-23 14:56:13 UTC
This will go out as an errata in the next update.



Note You need to log in before you can comment on or make changes to this bug.