Bug 1893594

Summary: CX-5 card: jumbo byte performance case run failed using openvswitch-2.9.7-1.el7fdn under 7.9
Product: Red Hat Enterprise Linux Fast Datapath Reporter: liting <tli>
Component: openvswitchAssignee: Timothy Redaelli <tredaelli>
openvswitch sub component: ovs-dpdk QA Contact: liting <tli>
Status: CLOSED NOTABUG Docs Contact:
Severity: unspecified    
Priority: unspecified CC: atragler, ctrautma, jhsiao, ktraynor, qding
Version: RHEL 7.7   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-10 02:00:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liting 2020-11-02 03:28:42 UTC
Description of problem:


Version-Release number of selected component (if applicable):
openvswitch-2.9.7-1.el7fdn
Rhel-7.9

How reproducible:

Steps to Reproduce:
1. in host run following ovs command 
/usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
/usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x2
/usr/bin/ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=4096,4096
/usr/bin/ovs-vsctl --timeout 10 add-br br0 -- set bridge br0 datapath_type=netdev
 /usr/bin/ovs-vsctl --timeout 10 set Open_vSwitch . other_config:pmd-cpu-mask=0x800000800000
/usr/bin/ovs-vsctl --timeout 10 add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:82:00.0 options:n_rxq=1 mtu_request=2000 options:n_rxq_desc=1024,n_txq_desc=1024
/usr/bin/ovs-vsctl --timeout 10 add-port br0 dpdk1 -- set Interface dpdk1 type=dpdk options:dpdk-devargs=0000:82:00.1 options:n_rxq=1 mtu_request=2000 options:n_rxq_desc=1024,n_txq_desc=1024
/usr/bin/ovs-vsctl --timeout 10 add-port br0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/var/run/openvswitch/dpdkvhostuserclient0 mtu_request=2000
/usr/bin/ovs-vsctl --timeout 10 add-port br0 dpdkvhostuserclient1 -- set Interface dpdkvhostuserclient1 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient1 options:vhost-server-path=/var/run/openvswitch/dpdkvhostuserclient1 mtu_request=2000

/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 del-flows br0
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow br0 idle_timeout=0,in_port=1,action=output:3
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow br0 idle_timeout=0,in_port=3,action=output:1
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow br0 idle_timeout=0,in_port=2,action=output:4
/usr/bin/ovs-ofctl -O OpenFlow13 --timeout 10 add-flow br0 idle_timeout=0,in_port=4,action=output:2

2. start guest.
/bin/bash -c "sudo -E taskset -c 3,5,29 /usr/libexec/qemu-kvm -m 8192 -smp 3,sockets=3,cores=1,threads=1 -cpu host,migratable=off -drive if=ide,file=rhel7.9-vsperf-1Q-noviommu.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -chardev socket,id=char0,path=/var/run/openvswitch/dpdkvhostuserclient0,server -netdev type=vhost-user,id=net1,chardev=char0,vhostforce,queues=1 -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=net1,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,rx_queue_size=1024,mq=on,vectors=4 -chardev socket,id=char1,path=/var/run/openvswitch/dpdkvhostuserclient1,server -netdev type=vhost-user,id=net2,chardev=char1,vhostforce,queues=1 -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=net2,csum=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off,rx_queue_size=1024,mq=on,vectors=4"

3. in guest run following testpmd command.
 sysctl vm.nr_hugepages=1
 mkdir -p /dev/hugepages
 mount -t hugetlbfs hugetlbfs /dev/hugepages
 cat /proc/meminfo
 rpm -ivh /root/dpdkrpms/dpdk*.rpm
 cat /proc/cmdline
 ifdown eth0
 ifdown eth1
 /usr/share/dpdk/usertools/dpdk-devbind.py --status
 /usr/share/dpdk/usertools/dpdk-devbind.py -u 00:03.0 00:04.0
 modprobe -r vfio
 modprobe -r vfio_iommu_type1
 modprobe vfio enable_unsafe_noiommu_mode=Y
 modprobe vfio-pci
 /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci 00:03.0 00:04.0
 /usr/share/dpdk/usertools/dpdk-devbind.py --status
 cd /usr/bin
 ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- --burst=64 -i --rxd=512 --txd=512 --nb-cores=2 --txq=1 --rxq=1 --max-pkt-len=2000
 testpmd> set fwd io
 testpmd> start

4. send traffic with trex.


Actual results:
result is 0. port cannot receive any traffic.
[root@dell-per730-56 ~]# ovs-ofctl dump-flows br0
 cookie=0x0, duration=167.982s, table=0, n_packets=0, n_bytes=0, in_port=1 actions=output:3
 cookie=0x0, duration=167.957s, table=0, n_packets=0, n_bytes=0, in_port=3 actions=output:1
 cookie=0x0, duration=167.933s, table=0, n_packets=0, n_bytes=0, in_port=4 actions=output:2
 cookie=0x0, duration=167.909s, table=0, n_packets=0, n_bytes=0, in_port=2 actions=output:4

guest testpmd output:
Telling cores to stop...

Waiting for lcores to finish...



  ---------------------- Forward statistics for port 0  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ----------------------------------------------------------------------------



  ---------------------- Forward statistics for port 1  ----------------------

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ----------------------------------------------------------------------------



  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++

  RX-packets: 0              RX-dropped: 0             RX-total: 0

  TX-packets: 0              TX-dropped: 0             TX-total: 0

  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Expected results:


Additional info:
job link:
https://beaker.engineering.redhat.com/jobs/4695627

[root@dell-per730-56 ~]# ethtool -i p4p2
driver: mlx5_core
version: 5.0-0
firmware-version: 16.27.2008 (MT_0000000012)
expansion-rom-version: 
bus-info: 0000:82:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
[root@dell-per730-56 ~]# lspci -s 0000:82:00.1
82:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]

Comment 1 liting 2020-11-10 02:00:33 UTC
I changed the trex sender version from v2.82 to v2.52, and rerun the case, it work well now. So it is not a bug. I will close it.