Description of problem: When Piranha detects a "split-brain" scenario it will not re-arp the VIP from the primary LVS director node. This is observed when running LVS in direct routing mode. It causes the communication to virtual IP address to go to the wrong node until the upstream router's ARP cache expires, which can be quite a long time (4 hours in some cases).
Created attachment 107236 [details] Patch fixing gratuitous ARP problem after scenario This patch fixes piranha so that gratuitous ARPs are sent after a split brain.
There is also a race condition, even with the above patch applied. It can be visualized in the following manner: Working case: Master Backup active ------> <------ inact active ------> <------ inact active -> Z Z <- inact active -> Z Z <- inact active -> Z (activate / send arps) Z <- active active -> Z arp... <------ active active ------> (shuts down) <------ inact Non-working case: the master doesn't send a new gratuitous ARP after the split-brain: Master Backup active ------> <------ inact active ------> <------ inact active -> Z Z <- inact active -> Z Z <- inact active -> Z (activate / send arps) Z <- active active -> Z Z <- active active ------> (shuts down) <------ inact So, we need to ensure that the backup always notifies the master that it is or was the master so that the master sends gratuitous arps for the virtual IP address. This can be done by using a "dying breath" heartbeat packet. When backup receives a heartbeat from the master, it can send one last ACTIVE packet prior to shutting down LVS services. This can cause the master to send gratuitous arps twice in some cases, but this should not cause problems: Master Backup <------ inact active ------> <------ inact active -> Z Z <- inact active -> Z Z <- inact active -> Z (activate / send arps) Z <- active active -> Z arp... <------ active active ------> } <------ dying "ACTIVE" packet } arp sent a second time arp... (shuts down) } <------ inact active ------>
Created attachment 107238 [details] Patch fixing race condition
The patch ensures that in the non-working case that at least one gratuitous ARP is sent for the virtual IP address; the fact that two are sent in the working case is a side effect. There are other ways to go about this; however, this is the simplest.
One way to reproduce the original split-brain is to simply 'killall -STOP pulse' on the master while monitoring /var/log/messages on the backup. When the backup takes over and sends ARPs, type 'killall -CONT pulse' on the master. Sometimes it will send arps; other times it won't.
Correction: The previous comment is how to recreate the race condition with relative ease.
This will go out as an errata in the next update.