Bug 2183002

Summary: balance-slb didn't work on pf/vf scenario with static link aggregation
Product: Red Hat Enterprise Linux Fast Datapath Reporter: mhou <mhou>
Component: openvswitch3.1Assignee: Mike Pattrick <mpattric>
Status: NEW --- QA Contact: mhou <mhou>
Severity: high Docs Contact:
Priority: high    
Version: RHEL 9.0CC: ctrautma, fleitner, jhsiao, kzhang, liali, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mhou 2023-03-30 07:55:56 UTC
Description of problem:
ping from bond brdige to peer side failed. Check the ovs log find below message:
2023-03-30T07:04:44.153Z|00062|bond(handler3)|DBG|Dropped 5 log messages in last 41 seconds (most recently, 18 seconds ago) due to excessive rate
2023-03-30T07:04:44.153Z|00063|bond(handler3)|DBG|bond balance-slb: member ens4f1v25: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Each PF creation can support the maximum number of vf. Select one of the vfs as slave of the bond. 
ip link set ens4f0 mtu 9200
ip link set ens4f1 mtu 9200
echo 64 > /sys/class/net/ens4f0/device/sriov_numvfs
echo 64 > /sys/class/net/ens4f1/device/sriov_numvfs
221: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9200 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:bd:1c:a4 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 8     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 9     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 10     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 11     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 12     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 13     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 14     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 15     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 16     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 17     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 18     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 19     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 20     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 21     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 22     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 23     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 24     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 25     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 26     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 27     link/ether 6e:9a:14:f9:7b:18 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust on
    vf 28     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 29     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 30     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 31     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 32     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 33     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 34     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 35     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 36     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 37     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 38     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 39     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 40     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 41     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 42     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 43     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 44     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 45     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 46     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 47     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 48     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 49     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 50     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 51     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 52     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 53     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 54     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 55     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 56     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 57     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 58     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 59     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 60     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 61     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 62     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 63     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    altname enp175s0f0

# ip link show ens4f1
222: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9200 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:bd:1c:a5 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 8     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 9     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 10     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 11     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 12     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 13     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 14     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 15     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 16     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 17     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 18     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 19     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 20     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 21     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 22     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 23     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 24     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 25     link/ether 02:f2:12:92:34:15 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust on
    vf 26     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 27     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 28     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 29     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 30     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 31     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 32     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 33     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 34     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 35     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 36     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 37     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 38     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 39     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 40     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 41     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 42     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 43     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 44     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 45     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 46     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 47     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 48     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 49     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 50     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 51     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 52     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 53     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 54     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 55     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 56     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 57     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 58     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 59     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 60     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 61     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 62     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 63     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    altname enp175s0f1

choose vf 25 from ens4f0 and vf 27 from ens4f1 as slave port.
ip link set ens4f0 vf 25 spoofchk off
ip link set ens4f0 vf 25 trust on
ip link set ens4f1 vf 27 spoofchk off
ip link set ens4f1 vf 25 trust on

2. On switch side, corresponding switch port of test PF wasn added in channel group without lacp.
sw-cisco9364# show interface brief 

--------------------------------------------------------------------------------
Port   VRF          Status IP Address                              Speed    MTU
--------------------------------------------------------------------------------
mgmt0  --           up     10.73.88.13                             1000    1500    
--------------------------------------------------------------------------------
Ethernet        VLAN    Type Mode   Status  Reason                 Speed     Port
Interface                                                                    Ch #
--------------------------------------------------------------------------------
.....(skip).......

Eth1/7/1        1000    eth  trunk  up      none                      25G(D) 64
Eth1/7/2        1000    eth  trunk  up      none                      25G(D) 64
------------------------------------------------------------------------------------------
sw-cisco9364# show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        b - BFD Session Wait
        S - Switched    R - Routed
        U - Up (port-channel)
        p - Up in delay-lacp mode (member)
        M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
64    Po64(SU)    Eth      NONE      Eth1/7/1(P)  Eth1/7/2(P)  

sw-cisco9364# show port-channel traffic interface port-channel 64
NOTE: Clear the port-channel member counters to get accurate statistics

ChanId      Port Rx-Ucst Tx-Ucst Rx-Mcst Tx-Mcst Rx-Bcst Tx-Bcst
------ --------- ------- ------- ------- ------- ------- -------
    64  Eth1/7/1    0.0%    0.0%  50.00%  48.93%    0.0%    0.0%
    64  Eth1/7/2    0.0%    0.0%  50.00%  51.06% 100.00%    0.0%

3. create a ovs bridge and attach vf port as slave interface for balance-slb.
systemctl restart openvswitch
ovs-appctl vlog/set console:lacp:dbg syslog:lacp:dbg file:lacp:dbg
ovs-appctl vlog/set console:bond:dbg syslog:bond:dbg file:bond:dbg
ovs-appctl vlog/set console:ovs_rcu:dbg syslog:ovs_rcu:dbg file:ovs_rcu:dbg
ovs-appctl vlog/set console:memory:dbg syslog:memory:dbg file:memory:dbg
ovs-vsctl set Open_vSwitch . other_config={}
ovs-vsctl --may-exist add-br bondbridge
ovs-vsctl set int bondbridge mtu_request=9200
ovs-ofctl mod-port bondbridge bondbridge up
ovs-vsctl add-bond bondbridge balance-slb ens4f1v25 ens4f0v27 lacp=off bond_mode=balance-slb  -- set Interface ens4f1v25 mtu_request=9200  -- set Interface ens4f0v27 mtu_request=9200
# ovs-vsctl show
d6152eff-ea84-4e6e-b452-4ad832befd72
    Bridge bondbridge
        Port balance-slb
            Interface ens4f1v25
            Interface ens4f0v27
        Port bondbridge
            Interface bondbridge
                type: internal
    ovs_version: "3.1.1"
# ovs-appctl bond/show
---- balance-slb ----
bond_mode: balance-slb
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
lb_output action: disabled, bond-id: -1
all members active: false
updelay: 0 ms
downdelay: 0 ms
next rebalance: 4012 ms
lacp_status: off
lacp_fallback_ab: false
active-backup primary: <none>
active member mac: 02:f2:12:92:34:15(ens4f1v25)

member ens4f0v27: enabled
  may_enable: true

member ens4f1v25: enabled
  active member
  may_enable: true


4. configure ip address on bond bridge and ping peer side
# ip addr show bondbridge
1523: bondbridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9200 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ba:32:1e:c1:71:40 brd ff:ff:ff:ff:ff:ff
    inet 172.31.152.210/24 scope global bondbridge
       valid_lft forever preferred_lft forever
    inet6 fe80::b832:1eff:fec1:7140/64 scope link 
       valid_lft forever preferred_lft forever
# ping -I bondbridge 172.31.152.1
PING 172.31.152.1 (172.31.152.1) from 172.31.152.210 bondbridge: 56(84) bytes of data.
^C
--- 172.31.152.1 ping statistics ---
12 packets transmitted, 0 received, 100% packet loss, time 11259ms

6. check ovs log as below:
2023-03-30T07:17:58.777Z|00086|bond(handler3)|DBG|Dropped 3 log messages in last 113 seconds (most recently, 80 seconds ago) due to excessive rate
2023-03-30T07:17:58.777Z|00087|bond(handler3)|DBG|bond balance-slb: member ens4f1v25: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-03-30T07:17:59.272Z|00018|bond(revalidator2)|DBG|bond balance-slb: member ens4f1v25: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-03-30T07:18:06.280Z|01371|bond|DBG|bond balance-slb: ens4f1v25 0kB (h76: 0kB + h227: 0kB), ens4f0v27 0kB
2023-03-30T07:18:16.292Z|01372|bond|DBG|bond balance-slb: ens4f1v25 0kB (h76: 0kB + h227: 0kB), ens4f0v27 0kB
2023-03-30T07:18:26.301Z|01373|bond|DBG|bond balance-slb: ens4f1v25 1kB (h76: 1kB + h227: 0kB), ens4f0v27 0kB
2023-03-30T07:18:36.312Z|01374|bond|DBG|bond balance-slb: ens4f1v25 0kB (h76: 0kB), ens4f0v27 0kB
2023-03-30T07:18:41.388Z|00088|bond(handler3)|DBG|Dropped 1 log messages in last 19 seconds (most recently, 19 seconds ago) due to excessive rate
2023-03-30T07:18:41.388Z|00089|bond(handler3)|DBG|bond balance-slb: member ens4f1v25: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off

7. peer side can capture icmp packets.
# tcpdump -i ens3f0 -ne
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens3f0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:19:40.525669 ba:32:1e:c1:71:40 > 40:a6:b7:3e:a5:60, ethertype IPv4 (0x0800), length 98: 172.31.152.210 > 172.31.152.1: ICMP echo request, id 549, seq 64, length 64
03:19:40.525685 40:a6:b7:3e:a5:60 > ba:32:1e:c1:71:40, ethertype IPv4 (0x0800), length 98: 172.31.152.1 > 172.31.152.210: ICMP echo reply, id 549, seq 64, length 64
03:19:41.549670 ba:32:1e:c1:71:40 > 40:a6:b7:3e:a5:60, ethertype IPv4 (0x0800), length 98: 172.31.152.210 > 172.31.152.1: ICMP echo request, id 549, seq 65, length 64
03:19:41.549684 40:a6:b7:3e:a5:60 > ba:32:1e:c1:71:40, ethertype IPv4 (0x0800), length 98: 172.31.152.1 > 172.31.152.210: ICMP echo reply, id 549, seq 65, length 64
03:19:42.573661 ba:32:1e:c1:71:40 > 40:a6:b7:3e:a5:60, ethertype IPv4 (0x0800), length 98: 172.31.152.210 > 172.31.152.1: ICMP echo request, id 549, seq 66, length 64
03:19:42.573668 40:a6:b7:3e:a5:60 > ba:32:1e:c1:71:40, ethertype IPv4 (0x0800), length 98: 172.31.152.1 > 172.31.152.210: ICMP echo reply, id 549, seq 66, length 64
03:19:43.597676 ba:32:1e:c1:71:40 > 40:a6:b7:3e:a5:60, ethertype IPv4 (0x0800), length 98: 172.31.152.210 > 172.31.152.1: ICMP echo request, id 549, seq 67, length 64
03:19:43.597683 40:a6:b7:3e:a5:60 > ba:32:1e:c1:71:40, ethertype IPv4 (0x0800), length 98: 172.31.152.1 > 172.31.152.210: ICMP echo reply, id 549, seq 67, length 64
03:19:44.621670 ba:32:1e:c1:71:40 > 40:a6:b7:3e:a5:60, ethertype IPv4 (0x0800), length 98: 172.31.152.210 > 172.31.152.1: ICMP echo request, id 549, seq 68, length 64
03:19:44.621677 40:a6:b7:3e:a5:60 > ba:32:1e:c1:71:40, ethertype IPv4 (0x0800), length 98: 172.31.152.1 > 172.31.152.210: ICMP echo reply, id 549, seq 68, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

On Dut side, Only icmp on tx can be capture
# tcpdump -i bondbridge
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bondbridge, link-type EN10MB (Ethernet), snapshot length 262144 bytes
03:20:15.340374 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 98, length 64
03:20:16.364374 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 99, length 64
03:20:17.388367 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 100, length 64
03:20:18.412375 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 101, length 64
03:20:19.436367 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 102, length 64
03:20:20.460372 IP hp-dl388g10-03.rhts.eng.pek2.redhat.com > 172.31.152.1: ICMP echo request, id 549, seq 103, length 64
Actual results:
1. ping peer side failed
2. check 'admissibility verdict is to drop pkt as different port is learned' in ovs log

Expected results:
1. ping peer side as well

Additional info:
If remove channel group from switch site(didn't used static link aggregation), icmp reply can be received as well.

Comment 1 Mike Pattrick 2023-06-08 15:21:45 UTC
I see:

> bond balance-slb: member ens4f1v25: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off

This indicates that OVS thinks it's getting a packet on the wrong port. This could be due to the switch, or the nic driver, or OVS.

You run "tcpdump -i bondbridge" but in this case I'd recommend "tcpdump -i ens4f1v25". Or whatever one leg of the bond member is. This would indicate if we actually do see the packets coming in on the correct leg.

I'll try and set up a reproducer, Intel nic right?

Comment 2 mhou 2023-06-14 12:30:35 UTC
Hello Mike

> I'll try and set up a reproducer, Intel nic right?

yep, I use xxv710 NIC. 

I try run same test on 5.14.0-318.el9.x86_64 + openvswitch3.1-3.1.0-24.el9fdp.x86_64 and still can see below message on ovs-vswitchd.log
# tail -f /var/log/openvswitch/ovs-vswitchd.log 
2023-06-14T12:22:50.559Z|00041|bridge|INFO|bridge guestbridge: using datapath ID 0000d629ba153d42
2023-06-14T12:22:50.559Z|00042|connmgr|INFO|guestbridge: added service controller "punix:/var/run/openvswitch/guestbridge.mgmt"
2023-06-14T12:22:50.624Z|00043|bridge|INFO|bridge guestbridge: added interface patchguest on port 1
2023-06-14T12:22:51.448Z|00044|bridge|INFO|bridge guestbridge: added interface 740ab41a0b764_l on port 2
2023-06-14T12:22:51.720Z|00045|bridge|INFO|bridge guestbridge: added interface c2d003d22df84_l on port 3
2023-06-14T12:22:51.973Z|00046|bridge|INFO|bridge guestbridge: added interface 65434c82dd954_l on port 4
2023-06-14T12:22:52.756Z|00047|connmgr|INFO|guestbridge<->unix#39: 1 flow_mods in the last 0 s (1 deletes)
2023-06-14T12:22:52.792Z|00048|connmgr|INFO|guestbridge<->unix#42: 1 flow_mods in the last 0 s (1 adds)
2023-06-14T12:22:58.775Z|00049|memory|INFO|172224 kB peak resident set size after 10.1 seconds
2023-06-14T12:22:58.775Z|00050|memory|INFO|handlers:3 idl-cells-Open_vSwitch:460 ports:7 revalidators:1 rules:10 udpif keys:21
2023-06-14T12:27:31.062Z|00051|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.063Z|00052|netdev_linux|WARN|ethtool command ETHTOOL_GSET on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.063Z|00053|bridge|INFO|bridge bondbridge: added interface ens4f1v11 on port 2
2023-06-14T12:27:31.063Z|00054|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.064Z|00055|netdev_linux|WARN|ethtool command ETHTOOL_GFLAGS on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.064Z|00056|netdev_linux|WARN|ethtool command ETHTOOL_GSET on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.064Z|00057|bridge|INFO|bridge bondbridge: added interface ens4f0v5 on port 3
2023-06-14T12:27:31.064Z|00058|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.064Z|00059|bond|INFO|member ens4f1v11: enabled
2023-06-14T12:27:31.064Z|00060|bond|INFO|member ens4f0v5: enabled
2023-06-14T12:27:31.065Z|00061|bond|INFO|bond balance-slb: active member is now ens4f0v5
2023-06-14T12:27:31.065Z|00062|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.065Z|00063|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.070Z|00064|netdev_linux|WARN|ethtool command ETHTOOL_GSET on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.070Z|00065|netdev_linux|WARN|ethtool command ETHTOOL_GSET on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.073Z|00066|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f1v11 failed: No such device
2023-06-14T12:27:31.073Z|00067|netdev_linux|WARN|ethtool command ETHTOOL_GDRVINFO on network device ens4f0v5 failed: No such device
2023-06-14T12:27:31.166Z|00068|bond|INFO|member ens4f0v5: link state down
2023-06-14T12:27:31.166Z|00069|bond|INFO|member ens4f0v5: disabled
2023-06-14T12:27:31.166Z|00070|bond|INFO|member ens4f1v11: link state down
2023-06-14T12:27:31.166Z|00071|bond|INFO|member ens4f1v11: disabled
2023-06-14T12:27:31.166Z|00072|bond|INFO|bond balance-slb: all members disabled
2023-06-14T12:27:31.270Z|00073|bond|INFO|member ens4f0v5: link state up
2023-06-14T12:27:31.271Z|00074|bond|INFO|member ens4f0v5: enabled
2023-06-14T12:27:31.271Z|00075|bond|INFO|member ens4f1v11: link state up
2023-06-14T12:27:31.271Z|00076|bond|INFO|member ens4f1v11: enabled
2023-06-14T12:27:31.271Z|00077|bond|INFO|bond balance-slb: active member is now ens4f0v5
2023-06-14T12:27:31.910Z|00001|bond(handler4)|DBG|bond balance-slb: member ens4f1v11: admissibility verdict is to drop pkt as different port is learned, active member: false, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:27:31.910Z|00001|bond(handler3)|DBG|bond balance-slb: member ens4f0v5: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:27:31.910Z|00002|bond(handler3)|DBG|bond balance-slb: member ens4f0v5: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:27:31.910Z|00003|bond(handler3)|DBG|bond balance-slb: member ens4f0v5: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:27:32.283Z|00001|bond(revalidator5)|DBG|bond balance-slb: member ens4f0v5: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:27:41.295Z|00078|bond|DBG|bond balance-slb: ens4f0v5 1kB (h40: 1kB), ens4f1v11 0kB
2023-06-14T12:27:51.696Z|00079|bond|DBG|bond balance-slb: ens4f0v5 2kB (h40: 2kB), ens4f1v11 0kB
2023-06-14T12:28:01.707Z|00080|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:28:11.720Z|00081|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:28:21.731Z|00082|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:28:31.743Z|00083|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:28:39.686Z|00002|bond(handler4)|DBG|Dropped 5 log messages in last 67 seconds (most recently, 15 seconds ago) due to excessive rate
2023-06-14T12:28:39.686Z|00003|bond(handler4)|DBG|bond balance-slb: member ens4f1v11: admissibility verdict is to drop pkt as different port is learned, active member: false, may_enable: true, enabled: true, LACP status: off
2023-06-14T12:28:41.754Z|00084|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:28:51.764Z|00085|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:29:01.778Z|00086|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:29:11.788Z|00087|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:29:21.799Z|00088|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB
2023-06-14T12:29:31.810Z|00089|bond|DBG|bond balance-slb: ens4f0v5 3kB (h40: 3kB), ens4f1v11 0kB


use tcpdump to capture packets from each slave interface.
# tcpdump -i ens4f0v5
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4f0v5, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:28:11.846088 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 912, seq 270, length 64
08:28:11.846117 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 912, seq 270, length 64
08:28:12.038082 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 913, seq 36, length 64
08:28:12.038108 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 913, seq 36, length 64
08:28:12.870082 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 912, seq 271, length 64
08:28:12.870109 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 912, seq 271, length 64
08:28:13.062084 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 913, seq 37, length 64
08:28:13.062111 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 913, seq 37, length 64
08:28:13.894081 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 912, seq 272, length 64
08:28:13.894101 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 912, seq 272, length 64
08:28:14.086084 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 913, seq 38, length 64
08:28:14.086104 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 913, seq 38, length 64
08:28:14.918081 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 912, seq 273, length 64
08:28:14.918102 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 912, seq 273, length 64
08:28:15.110085 IP 172.31.150.42 > 172.31.150.1: ICMP echo request, id 913, seq 39, length 64
08:28:15.110104 IP 172.31.150.1 > 172.31.150.42: ICMP echo reply, id 913, seq 39, length 64
08:28:15.442378 LLDP, length 270: sw-cisco9364


# tcpdump -i ens4f1v11
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens4f1v11, link-type EN10MB (Ethernet), snapshot length 262144 bytes
08:28:29.071321 LLDP, length 270: sw-cisco9364
08:28:34.073341 LLDP, length 270: sw-cisco9364
08:28:39.075216 LLDP, length 270: sw-cisco9364
08:28:39.686229 ARP, Reply 172.31.150.1 is-at 40:a6:b7:3e:a5:60 (oui Unknown), length 46
08:28:44.076963 LLDP, length 270: sw-cisco9364
08:28:49.079388 LLDP, length 270: sw-cisco9364
08:28:54.080940 LLDP, length 270: sw-cisco9364
08:28:59.082342 LLDP, length 270: sw-cisco9364
08:29:04.084252 LLDP, length 270: sw-cisco9364
08:29:09.086490 LLDP, length 270: sw-cisco9364
08:29:14.088800 LLDP, length 270: sw-cisco9364
08:29:15.290478 ARP, Request who-has 172.31.150.42 tell 172.31.150.1, length 46
08:29:19.090864 LLDP, length 270: sw-cisco9364
08:29:22.694129 ARP, Reply 172.31.150.1 is-at 40:a6:b7:3e:a5:60 (oui Unknown), length 46
08:29:24.092902 LLDP, length 270: sw-cisco9364
08:29:29.094960 LLDP, length 270: sw-cisco9364
08:29:34.096839 LLDP, length 270: sw-cisco9364
08:29:39.098052 LLDP, length 270: sw-cisco9364
08:29:44.100178 LLDP, length 270: sw-cisco9364
08:29:49.101688 LLDP, length 270: sw-cisco9364
08:29:54.103838 LLDP, length 270: sw-cisco9364
08:29:59.105299 LLDP, length 270: sw-cisco9364
08:30:02.118068 IP6 hp-dl388g10-03.rhts.eng.pek2.redhat.com > ff02::2: ICMP6, router solicitation, length 16
08:30:04.106882 LLDP, length 270: sw-cisco9364
08:30:04.442470 ARP, Request who-has 172.31.150.42 tell 172.31.150.1, length 46

Comment 3 mhou 2023-07-16 10:32:16 UTC
smae issue was found on juniper switch and cx5 nic.

[root@dell-per740-09 ovs_bond_function]# ethtool -i ens3f0np0
driver: mlx5_core
version: 5.14.0-333.el9.x86_64
firmware-version: 26.33.1048 (MT_0000000531)
expansion-rom-version: 
bus-info: 0000:5e:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@dell-per740-09 ovs_bond_function]# ethtool -i ens3f1np1
driver: mlx5_core
version: 5.14.0-333.el9.x86_64
firmware-version: 26.33.1048 (MT_0000000531)
expansion-rom-version: 
bus-info: 0000:5e:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes


# ovs-appctl bond/show
---- balance-slb ----
bond_mode: balance-slb
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
lb_output action: disabled, bond-id: -1
all members active: false
updelay: 0 ms
downdelay: 0 ms
next rebalance: 9967 ms
lacp_status: off
lacp_fallback_ab: false
active-backup primary: <none>
active member mac: b8:3f:d2:43:28:94(ens3f0np0)

member ens3f0np0: enabled
  active member
  may_enable: true
  hash 4: 15159948 kB load

member ens3f1np1: enabled
  may_enable: true
  hash 24: 15249907 kB load



2023-07-16T10:29:49.853Z|00067|bond|DBG|bond balance-slb: ens3f0np0 0kB (h4: 0kB + h24: 0kB + h108: 0kB + h223: 0kB), ens3f1np1 0kB (h34: 0kB)
2023-07-16T10:29:59.863Z|00068|bond|DBG|bond balance-slb: ens3f0np0 0kB (h4: 0kB + h24: 0kB + h108: 0kB + h223: 0kB), ens3f1np1 0kB (h34: 0kB)
2023-07-16T10:30:09.874Z|00069|bond|DBG|bond balance-slb: ens3f0np0 0kB (h108: 0kB + h4: 0kB + h24: 0kB + h223: 0kB), ens3f1np1 0kB (h34: 0kB)
2023-07-16T10:30:10.392Z|00002|bond(handler1)|DBG|bond balance-slb: member ens3f0np0: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-07-16T10:30:10.881Z|00002|bond(revalidator7)|DBG|bond balance-slb: member ens3f0np0: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-07-16T10:30:15.858Z|00001|bond(handler8)|DBG|bond balance-slb: member ens3f0np0: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-07-16T10:30:15.892Z|00003|bond(revalidator7)|DBG|bond balance-slb: member ens3f0np0: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-07-16T10:30:19.897Z|00070|bond|DBG|bond balance-slb: ens3f0np0 0kB (h4: 0kB + h24: 0kB + h223: 0kB + h108: 0kB), ens3f1np1 0kB
2023-07-16T10:30:29.908Z|00071|bond|DBG|bond balance-slb: ens3f0np0 7154567kB (h4: 3783174kB + h24: 3371392kB + h223: 0kB + h108: 0kB), ens3f1np1 0kB (h34: 0kB)
2023-07-16T10:30:29.908Z|00072|bond|INFO|bond balance-slb: shift 3371392kB of load (with hash 24) from ens3f0np0 to ens3f1np1 (now carrying 3783174kB and 3371392kB load, respectively)
2023-07-16T10:30:39.918Z|00073|bond|DBG|bond balance-slb: ens3f0np0 17554033kB (h4: 17554033kB + h223: 0kB + h108: 0kB), ens3f1np1 16632498kB (h24: 16632498kB + h34: 0kB)
2023-07-16T10:30:43.161Z|00003|bond(handler1)|DBG|Dropped 22 log messages in last 27 seconds (most recently, 13 seconds ago) due to excessive rate
2023-07-16T10:30:43.161Z|00004|bond(handler1)|DBG|bond balance-slb: member ens3f0np0: admissibility verdict is to drop pkt as different port is learned, active member: true, may_enable: true, enabled: true, LACP status: off
2023-07-16T10:30:49.930Z|00074|bond|DBG|bond balance-slb: ens3f0np0 23941097kB (h4: 23941097kB + h223: 0kB + h108: 0kB), ens3f1np1 23878170kB (h24: 23878170kB + h34: 0kB)