Bug 1893610

Description of problem:


Version-Release number of selected component (if applicable):
openvswitch-2.9.7-1.el7fdn
Rhel-7.9

How reproducible:


Steps to Reproduce:
1. In host run following command.
ovs-vsctl add-br br0
ovs-vsctl add-port br0 enp5s0np0
ovs-vsctl add-port br0 enp5s0np1
ovs-vsctl add-port br0 tap0
ovs-vsctl add-port br0 tap1
ovs-ofctl -O OpenFlow13 add-flow br0 idle_timeout=0,in_port=1,action=output:2
ovs-ofctl -O OpenFlow13 add-flow br0 idle_timeout=0,in_port=2,action=output:1
ovs-ofctl -O OpenFlow13 add-flow br0 idle_timeout=0,in_port=3,action=output:4
ovs-ofctl -O OpenFlow13 add-flow br0 idle_timeout=0,in_port=4,action=output:3

2. start guest
 /bin/bash -c "sudo -E taskset -c 2,4,32 /usr/libexec/qemu-kvm -m 8192 -smp 3 -cpu host,migratable=off -drive if=ide,file=rhel7.9-vsperf-1Q-viommu.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -M q35,kernel-irqchip=split -device intel-iommu,device-iotlb=on,intremap,caching-mode=true -device pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2 -device pcie-root-port,id=root.3,slot=3 -device pcie-root-port,id=root.4,slot=4 -netdev type=tap,id=eth0,script=no,downscript=no,ifname=tap0,vhost=on -device virtio-net-pci,mac=00:00:00:00:00:01,iommu_platform=on,ats=on,bus=root.2,netdev=eth0,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -netdev type=tap,id=eth1,script=no,downscript=no,ifname=tap1,vhost=on -device virtio-net-pci,mac=00:00:00:00:00:02,iommu_platform=on,ats=on,bus=root.3,netdev=eth1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off"

3. In guest, run following command.
 sysctl vm.nr_hugepages=1
 mkdir -p /dev/hugepages
 mount -t hugetlbfs hugetlbfs /dev/hugepages
 cat /proc/meminfo
 rpm -ivh /root/dpdkrpms/dpdk*.rpm
 cat /proc/cmdline
 ifdown eth0
 ifdown eth1
 /usr/share/dpdk/usertools/dpdk-devbind.py --status
 /usr/share/dpdk/usertools/dpdk-devbind.py -u 02:00.0 03:00.0
 modprobe vfio
 modprobe vfio-pci
 /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci 02:00.0 03:00.0
 /usr/share/dpdk/usertools/dpdk-devbind.py --status
 cd /usr/bin
 ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- --burst=64 -i --rxd=512 --txd=512 --nb-cores=2 --txq=1 --rxq=1
 testpmd> set fwd io
 testpmd> start

4. Use T-rex send traffic

Actual results:
Only one nfp port enp5s0np0 can receive packet. The other nfp port enp5s0np1 cannot receive any packet. The performance got 0. 

[root@dell-per730-52 ~]# ovs-ofctl dump-flows br0
 cookie=0x0, duration=71.007s, table=0, n_packets=0, n_bytes=0, in_port=tap1 actions=output:enp5s0np1
 cookie=0x0, duration=67.443s, table=0, n_packets=8553825, n_bytes=513229500, in_port=enp5s0np1 actions=output:tap1
 cookie=0x0, duration=64.340s, table=0, n_packets=8553790, n_bytes=513227400, in_port=tap0 actions=output:enp5s0np0
 cookie=0x0, duration=60.714s, table=0, n_packets=0, n_bytes=0, in_port=enp5s0np0 actions=output:tap0

guest testpmd result.
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 8868672        TX-dropped: 0             TX-total: 8868672
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 8868672        RX-dropped: 0             RX-total: 8868672
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 8868672        RX-dropped: 0             RX-total: 8868672
  TX-packets: 8868672        TX-dropped: 0             TX-total: 8868672
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


Expected results:
The case can run successfully on nfp card.

Additional info:
beaker job:
https://beaker.engineering.redhat.com/jobs/4689908

[root@dell-per730-52 ~]# ethtool -i enp5s0np0
driver: nfp
version: 3.10.0-1160.el7.x86_64 SMP mod_
firmware-version: 0.0.3.5 0.22 nic-2.1.16.1 nic
expansion-rom-version: 
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no