Bug 586557

I'm guessing there's no guarantee that both nics will be involved for a single transfer. A better test would be to create enough network traffic that you need the bandwidth from both, and then check that they both are handling the traffic. But I'll pass this over to kernel for the bonding guys to comment.

There is no guarantee that traffic will flow on both bonding interfaces.  A hash based on the contents of the frame is computed and used to determine output port selection.  The default hash policy only examines Layer 2 (source and destination MAC addresses) data in the frame.  If all traffic is destined for a single-host or multiple hosts on a separate network connected via a router, only one link will be used when using Layer 2 hashing.  For that reason, I would suggest using Layer2+3 or Layer3+4 hashing instead depending on your traffic pattern.

I would suggest adding the option:

"xmit_hash_policy=layer2+3"

to your bonding options.

More details about these options can be found in the bonding documentation about the xmit_hash_policy option:

xmit_hash_policy

        Selects the transmit hash policy to use for slave selection in
        balance-xor and 802.3ad modes.  Possible values are:

        layer2

                Uses XOR of hardware MAC addresses to generate the
                hash.  The formula is

                (source MAC XOR destination MAC) modulo slave count

                This algorithm will place all traffic to a particular
                network peer on the same slave.

                This algorithm is 802.3ad compliant.

        layer2+3

                This policy uses a combination of layer2 and layer3
                protocol information to generate the hash.

                Uses XOR of hardware MAC addresses and IP addresses to
                generate the hash.  The formula is

                (((source IP XOR dest IP) AND 0xffff) XOR
                        ( source MAC XOR destination MAC ))
                                modulo slave count

                This algorithm will place all traffic to a particular
                network peer on the same slave.  For non-IP traffic,
                the formula is the same as for the layer2 transmit
                hash policy.

                This policy is intended to provide a more balanced
                distribution of traffic than layer2 alone, especially
                in environments where a layer3 gateway device is
                required to reach most destinations.

                This algorithm is 802.3ad compliant.

        layer3+4

                This policy uses upper layer protocol information,
                when available, to generate the hash.  This allows for
                traffic to a particular network peer to span multiple
                slaves, although a single connection will not span
                multiple slaves.

                The formula for unfragmented TCP and UDP packets is

                ((source port XOR dest port) XOR
                         ((source IP XOR dest IP) AND 0xffff)
                                modulo slave count

                For fragmented TCP or UDP packets and all other IP
                protocol traffic, the source and destination port
                information is omitted.  For non-IP traffic, the
                formula is the same as for the layer2 transmit hash
                policy.

                This policy is intended to mimic the behavior of
                certain switches, notably Cisco switches with PFC2 as
                well as some Foundry and IBM products.

                This algorithm is not fully 802.3ad compliant.  A
                single TCP or UDP conversation containing both
                fragmented and unfragmented packets will see packets
                striped across two interfaces.  This may result in out
                of order delivery.  Most traffic types will not meet
                this criteria, as TCP rarely fragments traffic, and
                most UDP traffic is not involved in extended
                conversations.  Other implementations of 802.3ad may
                or may not tolerate this noncompliance.

        The default value is layer2.  This option was added in bonding
        version 2.6.3.  In earlier versions of bonding, this parameter
        does not exist, and the layer2 policy is the only policy.  The
        layer2+3 value was added for bonding version 3.2.2.

I would also suggest adding these options to ifcfg-bond0:

BONDING_OPTS="miimon=80 mode=4 xmit_hash_policy=layer2+3"

and removing this line and only this line from modprobe.conf:

options bond0 miimon=80 mode=4

Each time someone has asked about this issue, it turns out to be a configuration issue, so I am going to close this bug.  Feel free to re-open it if you are still having problems after switching to layer2+3 or layer3+4 and provide more details about the traffic being transmitted by the bond.