Bug 441716

Matsuya-san,
     We already put in a different patch to 5.2 that should significantly reduce
the ARP delay.  Please have them test with the 5.2 latest snapshots and see if
they see better results (you should see approximately 1-2 seconds downtime after
migration).  Also, I'm not sure what the new feature you are talking about is;
live migration has been supported since 5.0, so this is just a bug that hasn't
been fixed up until this point.

Thanks,
Chris Lalancette

The problem with the original patch is that it put in an Xen-specific solution
to a generic networking problem.  That is why it wasn't taken since we already
had the generic upstream patch.

Now unfortunately it appears that the Xen netfront driver in RHEL5 does not send
the gratuitous ARP packet at the correct time which is a separate issue that was
not identified in time.

So what we will do is fix the second issue as part of RHEL 5.3 and zstream.

Created attachment 302697 [details]
[XEN] netfront: Send fake arp when link gets carrier

As it is the Xen netfront driver will transmit a fake ARP when the link gets an
IP address and when the link is brought up administratively.  However, this
overlooks the case when the first two events occur without a link carrier. 
Thus to be sure that the packet makes it out we also need to attempt a transmit
when the carrier comes up which can be detected through the NETDEV_CHANGE
event.

This is what this patch does.

Created attachment 303426 [details]
Send fake ARP when link gets carrier

The patch posted didn't really work because of a few silly cut-n-paste errors. 
Here's a patch that actually does work, for me anyway.

in kernel-2.6.18-93.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Created attachment 310783 [details]
Delay carrier on until dom0 has carrier.

I've looked at it further and in fact both upstream and RHEL5 are still subject
to  a race between domU and dom0.  The problem is that when the ARP packet in
domU is sent dom0's backend interface may not be ready yet.  In particular, the
backend's carrier flag may not have come up yet which means that the bridge
won't let it transmit any packets since it is yet to enter the forwarding
state.

Here is a patch that attempts to delay domU from sending any packets until dom0
is ready.

Note that even this isn't perfect since dom0's bridge might have further
unpredictable delays in it.  But in the long term all this should get folded in
to dom0 anyway since only it is in a position to do this reliably.  As it is
this patch implicitly adds a delay in domU too (through the link watch layer)
which should counter-act the delay in dom0 (Ick!).

Created attachment 310792 [details]
Only delay grat ARP

Unfortunately delaying the carrier on event doesn't quite work since the rest
of the driver requires it to function.	This patch simply delays the fake ARP
instead which works for me.  But please test this on your machines since I
never saw the race on my machines anyway.

Created attachment 312200 [details]
[BRIDGE]: eliminate workqueue for carrier check

This upstream patch eliminates the delay.  So if your testing confirms my guess
then all we have to do is merge this patch.  My testing on Hari's machines also
shows the previous patch isn't necessary (I wasn't able to reproduce the race
locally).

Author: Stephen Hemminger <shemminger>
Date:	Thu Feb 22 01:10:18 2007 -0800

    [BRIDGE]: eliminate workqueue for carrier check

    Having a work queue for checking carrier leads to lots of race issues.
    Simpler to just get the cost when data structure is created and
    update on change.

    Signed-off-by: Stephen Hemminger <shemminger>
    Signed-off-by: David S. Miller <davem>

Created attachment 313096 [details]
[BRIDGE]: eliminate workqueue for carrier check

Just as before this patch is only needed in dom0.  The two pervious domU
patches are still required.

This is a back-port of two upstream patches:

commit 269def7c505b4d229f9ad49bf88543d1e605533e
Author: Stephen Hemminger <shemminger>
Date:	Thu Feb 22 01:10:18 2007 -0800

    [BRIDGE]: eliminate workqueue for carrier check

    Having a work queue for checking carrier leads to lots of race issues.
    Simpler to just get the cost when data structure is created and
    update on change.

    Signed-off-by: Stephen Hemminger <shemminger>
    Signed-off-by: David S. Miller <davem>

and

commit de79059ecd7cd650f3788ece978a64586921d1f1
Author: Aji Srinivas <emc.com>
Date:	Wed Mar 7 16:10:53 2007 -0800

    [BRIDGE]: adding new device to bridge should enable if up

    One change introduced by the workqueue removal patch is that adding an
    interface that is up to a bridge which is also up does not ever call
    br_stp_enable_port(), leaving the port in DISABLED state until we do
    ifconfig down and up or link events occur.

    The following patch to the br_add_if function fixes it.
    This is a regression introduced in 2.6.21.

    Submitted-by: Aji_Srinivas
    Signed-off-by: Stephen Hemminger <shemminger>
    Signed-off-by: David S. Miller <davem>

Created attachment 313197 [details]
[BRIDGE]: eliminate workqueue for carrier check

Cool, I just noticed that this patch actually fixes an unrelated (and critical)
bug too in RHEL5.  So we can push this regardless of the Xen issue.

I think this has been an elusive problem and some earlier fixes ended up not solving the issue completely or in all cases. But Herbert can answer better.
Putting this into needinfo for Herbert.

This bug has been marked for inclusion in the Red Hat Enterprise Linux 5.3
Release Notes.

To aid in the development of relevant and accurate release notes, please fill
out the "Release Notes" field above with the following 4 pieces of information:


Cause:   What actions or circumstances cause this bug to present.

Consequence:  What happens when the bug presents.

Fix:   What was done to fix the bug.

Result:  What now happens when the actions or circumstances above occur. (NB:
this is not the same as 'the bug doesn't present anymore')

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-0225.html

*** Bug 622574 has been marked as a duplicate of this bug. ***