Bug 560588 - Adding bonding in balance-alb mode to bridge causes host network connectivity to be lost [rhel-5.4.z]
Adding bonding in balance-alb mode to bridge causes host network connectivity...
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel (Show other bugs)
All Linux
low Severity medium
: rc
: ---
Assigned To: Jiri Pirko
Red Hat Kernel QE team
: ZStream
Depends On: 487763
  Show dependency treegraph
Reported: 2010-02-01 04:25 EST by RHEL Product and Program Management
Modified: 2016-04-26 09:31 EDT (History)
15 users (show)

See Also:
Fixed In Version: kernel-2.6.18-164.13.1.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2010-03-16 20:20:07 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0147 normal SHIPPED_LIVE Important: kernel security and bug fix update 2010-03-16 20:18:44 EDT

  None (edit)
Description RHEL Product and Program Management 2010-02-01 04:25:46 EST
This bug has been copied from bug #487763 and has been proposed
to be backported to 5.4 z-stream (EUS).
Comment 3 Jiri Pirko 2010-02-10 02:39:49 EST
in 2.6.18-164.13.1.el5
Comment 8 errata-xmlrpc 2010-03-16 20:20:07 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

Comment 9 imarlier@brightcove.com 2010-03-24 13:34:06 EDT
Though my understanding is that this fix should have been included with the 2.6.18-164.15.1 kernel update (based on the ERRATA notice above), I am still experiencing this issue.

The problem that I am having is exactly as described in https://bugzilla.redhat.com/show_bug.cgi?id=487763 (Comment #3, for example)

The setup I am testing is the following:
* 4 physical interfaces (eth0 - eth3) are bonded using balance-alg.
* tagged interfaces are created from the resulting bond0 (bond0.1, bond0.2, etc)
* bridges are created for each of those tagged interfaces (xenbr0.1, xenbr0.2, etc), and the bonds are added to those bridges.

My understanding is that each of the bridges should end up having all of the physical interface MAC addresses marked local, however only the master interface on the bond is marked as local, as shown below (all 4 interfaces in the below are physically located on the machine).

[root@prickle1 ~]# uname -a
Linux prickle1 2.6.18-164.15.1.el5xen #1 SMP Wed Mar 17 12:04:23 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
[root@prickle1 ~]# brctl showmacs xenbr0.1 | grep "a4:ba:db:09:2c:"
  1     a4:ba:db:09:2c:1c       yes                0.00
  1     a4:ba:db:09:2c:1e       no               111.87
  1     a4:ba:db:09:2c:84       no               111.87
  1     a4:ba:db:09:2c:86       no               111.87

I am able to do some testing/alterations to the machines that are having this issue, if there's something that I can do to either fix this or help with testing.  

Please also let me know any other information that would be helpful.
Comment 10 Jiri Pirko 2010-03-24 14:59:54 EDT
This looks like a different issue to me (vlans are involved). Please fill up new bz for this, and put me in cc. Thanks.
Comment 11 Andy Gospodarek 2010-03-24 15:18:54 EDT
The VLANs appears to be irrelevant.  

This appears to be exactly the same issue described in bug 506770 which we have closed as a WONTFIX for the time being.  I would strongly recommend switching to mode 5 when using bonding + bridging in a virt environment.
Comment 12 imarlier@brightcove.com 2010-03-24 15:49:03 EDT
Thanks for the recommendation, Andy.  I'll look into switching to mode 5 in this environment.

Bug 506770 appears to be offlimits (I get permission denied when I attempt to view it); is there a way that I might be able to get access to it to verify that it is, in fact, the same issue?
Comment 13 Andy Gospodarek 2010-03-29 08:49:49 EDT
(In reply to comment #12)
> Thanks for the recommendation, Andy.  I'll look into switching to mode 5 in
> this environment.
> Bug 506770 appears to be offlimits (I get permission denied when I attempt to
> view it); is there a way that I might be able to get access to it to verify
> that it is, in fact, the same issue?    

Sorry I haven't gotten back to you on this.  I can try to open it up, but I suspect I might not be able to make that happen.  If I cannot I'll post the relevant comments here.
Comment 14 Andy Gospodarek 2010-03-29 14:38:10 EDT

The fundamental problem described in bug 506770 is that ALB mode's receive load balancing (RLB) is designed to function as if the bond device is an end-point, not a device that will be responsible for accepting traffic that will need to be forwarded to other devices.

(This is a bit confusing and complex, so for clarity I will use 'bridge' to mean the bridge internal to the host that contains the bond and guest interfaces and I will use 'switch' to mean another device that is connected to the slaves in the bond.)

The patch added in this bug addresses the issue where traffic will be destined for a MAC address that belongs to one of the slaves in the bond, so it will not be properly received by the bridge (since the bridge will have the same MAC address as the first member of the bond).  This patch rewrites the frames as they come into the host and puts the mac address of the bond as the destination MAC.  This will resolve the case where the bridge that has an IP address will not maintain connectivity, but it will not prevent hosts connected via other members of the bridge (like veth0) to maintain connectivity (what bug 506770 was about).

Right now, the RLB code keeps track of all hosts that have sent frames on the bond recently (in the last 2 seconds) and attempts to balance the traffic by sending gratuitous ARPs with the source MAC address of the transmitting slave if balancing is needed.  All ARP responses will also have the MAC of the transmitting slave as the source MAC for ARP traffic coming out of the bond.

You can quickly see how the MAC of a guest may be overwritten when the bonding code starts controlling ARP traffic.  Once other hosts receive the slave device's MAC addresses as the MAC for the guest's IP, connectivity will be lost until the guest starts sending ARP frames again.  I created some patches that would allow the RLB code to maintain knowledge of the guest's MACs and not overwrite them with the slaves' MAC addresses when sending ARPs, but this still did not completely address the problem.  The possibility still exists that a single broadcast (unlearned or directed) frame could result in a loss of connectivity to the guest until the guest begins sending ARP requests.

The fact that there is no easy way to dynamically tell the bridge what addresses are considered local (the problem you are seeing), and there is no way to easily tell the bridge what guest MAC addresses should not be relearned on the bond0 interface in the bridge, supporting ALB mode is not possible.  It may be possible to write some ebtables rules to make it work, but that is not a dynamic or scalable solution.

Note You need to log in before you can comment on or make changes to this bug.