Description of problem:

Port state changes to "NO-CARRIER" on the back to back connected peer system if E810 binds vfio-pci. 


Version-Release number of selected component (if applicable):

# rpm -qa | grep dpdk
dpdk-19.11-5.el8_2.x86_64
dpdk-tools-19.11-5.el8_2.x86_64

and also on version dpdk/19.11/4.el8fdb.1


How reproducible:

Topo:

  System_1 (E810, vfio-pci)   <---->  System_2

In my environment, System_1 ens1f0 (E810 port1) back to back connect to  System_2 ens2f0 (mlx5_core port1) and System_1 ens1f1 <----> System_2 ens2f1.





Steps to Reproduce:

~~~ 1. Initial state

The E810 port ens1f0 at 0000:3b:00.0 uses kernel driver.

System_1# dpdk-devbind --status

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection 1521' if=eno3 drv=igb unused= *Active*
0000:01:00.1 'I350 Gigabit Network Connection 1521' if=eno4 drv=igb unused= 
0000:19:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eno1 drv=ixgbe unused= 
0000:19:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' if=eno2 drv=ixgbe unused= 
0000:3b:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens1f0 drv=ice unused= 
0000:3b:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens1f1 drv=ice unused= 
0000:5e:00.0 'MT27800 Family [ConnectX-5] 1017' if=ens2f0 drv=mlx5_core unused= 
0000:5e:00.1 'MT27800 Family [ConnectX-5] 1017' if=ens2f1 drv=mlx5_core unused= 


System_1 E810 port ens1f0 is up

System_1#ip link
...
3: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:18:fc:d8 brd ff:ff:ff:ff:ff:ff


The peer System_2 ens2f0 is up

System_2#ip link
...
7: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:8e:0c:18 brd ff:ff:ff:ff:ff:ff



~~~ 2.Bind vfio-pci to E810

On System_1

System_1# modprobe vfio-pci
System_1# modprobe vfio
System_1# dpdk-devbind -b vfio-pci 0000:3b:00.0

System_1# dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:3b:00.0 'Ethernet Controller E810-C for QSFP 1592' drv=vfio-pci unused=ice

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection 1521' if=eno3 drv=igb unused=vfio-pci *Active*
...


On System_2

System_2#ip link
...
7: ens2f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:8e:0c:18 brd ff:ff:ff:ff:ff:ff



~~~ 3. Unbind E810 port from vfio-pci and goes back to kernel driver(ice), then the peer port on System_2 comes up.
 
System_1# driverctl unset-override 0000:3b:00.0

System_2# ip link
...
7: ens2f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 98:03:9b:8e:0c:18 brd ff:ff:ff:ff:ff:ff


Actual results:

 E810's peer port shows "NO-CARRIER".

Expected results:

 E810's peer port shows up without change when vfio-pci is bound.
 


Additional info:

1. Same issue on another port of E810 card. When ens1f1 binds vfio-pci, its peer ports ens2f1 becomes "NO-CARRIER".

2. We tried to start a testpmd cross ens1f0 and ens1f1 (on System_1) when vfio-pci bound on these two ports. The peer ports ens2f0 and ens2f1 came up, and pktgen on System_2 was working fine.

3. Tested on i40e card, the issue was not seen.