Description of problem: [root@localhost ~]# -iot@localhost ~]# testpmd -l 0-8 -n 4 -w 03:00.0 -w 04:00.0 -- --rxq=2 --txq=2 EAL: Detected 9 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Probing VFIO support... EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: PCI device 0000:03:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:101a net_mlx5 net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration EAL: PCI device 0000:04:00.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 15b3:101a net_mlx5 net_mlx5: MPLS over GRE/UDP tunnel offloading disabled due to old OFED/rdma-core version or firmware configuration Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=211456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) [ 674.840044] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) [ 680.695218] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) [ 680.697080] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) [ 686.473923] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) [ 692.366890] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) [ 692.368550] infiniband mlx5_0: mlx5_ib_post_send_wait:919:(pid 2105): reg umr failed (5) Signal 2 received, preparing to exit... LATENCY_STATS: failed to remove Rx callback for pid=0, qid=0 LATENCY_STATS: failed to remove Rx callback for pid=0, qid=1 LATENCY_STATS: failed to remove Tx callback for pid=0, qid=0 LATENCY_STATS: failed to remove Tx callback for pid=0, qid=1 Shutting down port 0... Stopping ports... Done Closing ports... Port 0 is now not stopped Done Shutting down port 1... Stopping ports... Done Closing ports... Done Bye... [root@localhost ~]# Version-Release number of selected component (if applicable): [root@localhost ~]# rpm -qa | grep dpdk dpdk-tools-18.11-4.el7_6.x86_64 dpdk-18.11-4.el7_6.x86_64 [root@localhost ~]# [root@localhost ~]# uname -r 3.10.0-1062.el7.x86_64 [root@localhost ~]# same version in host How reproducible: 100% Steps to Reproduce: 1. create guest virsh create g1.xml 2. create VFs on dual ports PFs echo 1 > /sys/class/net/p6p1/device/sriov_numvfs echo 1 > /sys/class/net/p6p2/device/sriov_numvfs 3. attach VF to VM virsh attach-device g1 vf1.xml virsh attach-device g1 vf2.xml 4. run testpmd in guest Actual results: Expected results: Additional info: - boot cmdline in host: [root@dell-per730-04 perf]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-1101.el7.x86_64 root=/dev/mapper/rhel_dell--per730--04-root ro intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=16 ksdevice=bootif crashkernel=auto rd.lvm.lv=rhel_dell-per730-04/root rd.lvm.lv=rhel_dell-per730-04/swap console=ttyS0,115200n81 [root@dell-per730-04 perf]# - boot cmdline in guest: [root@localhost ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-1062.el7.x86_64 root=/dev/mapper/rhel-root ro intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=1 rhgb console=ttyS0,115200 crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS0 [root@localhost ~]# Same issue with two cards: [root@dell-per730-04 perf]# lspci -m -s 04:00.0 04:00.0 "Ethernet controller" "Mellanox Technologies" "MT28800 Family [ConnectX-5 Ex]" "Mellanox Technologies" "Device 0002" [root@dell-per730-04 perf]# [root@dell-per730-04 perf]# ethtool -i p6p1 driver: mlx5_core version: 5.0-0 firmware-version: 16.26.1040 (MT_0000000009) expansion-rom-version: bus-info: 0000:04:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes [root@dell-per730-04 perf]# [root@dell-per730-05 perf]# lspci -m -s 84:00.0 84:00.0 "Ethernet controller" "Mellanox Technologies" "MT27800 Family [ConnectX-5]" "Mellanox Technologies" "Device 0007" [root@dell-per730-05 perf]# [root@dell-per730-05 perf]# ethtool -i p2p1 driver: mlx5_core version: 5.0-0 firmware-version: 16.26.1040 (MT_0000000012) expansion-rom-version: bus-info: 0000:84:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes [root@dell-per730-05 perf]#
Hi qding, does this still reproduce? if so, can you please provide a setup with reproduction steps? Thanks, Alaa
Created attachment 1657761 [details] XML for creating guest
Created attachment 1657762 [details] XML for attaching VF1 to guest
Created attachment 1657766 [details] XML for attaching VF2 to guest
(In reply to Alaa Hleihel (Mellanox) from comment #3) > Hi qding, > > does this still reproduce? > if so, can you please provide a setup with reproduction steps? > 1. set bootoption in host as below and reboot system if anything changed [root@dell-per740-10 ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-1125.el7.x86_64 root=/dev/mapper/rhel_dell--per740--10-root ro crashkernel=auto intel_iommu=on iommu=pt default_hugepagesz=1GB hugepagesz=1G hugepages=16 spectre_v2=retpoline rd.lvm.lv=rhel_dell-per740-10/root rd.lvm.lv=rhel_dell-per740-10/swap console=ttyS0,115200n81 [root@dell-per740-10 ~]# 2. install required packages wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-img-rhev-2.12.0-33.el7_7.8.x86_64.rpm wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-kvm-common-rhev-2.12.0-33.el7_7.8.x86_64.rpm wget http://download-node-02.eng.bos.redhat.com/brewroot/packages/qemu-kvm-rhev/2.12.0/33.el7_7.8/x86_64/qemu-kvm-rhev-2.12.0-33.el7_7.8.x86_64.rpm yum -y install libvirt yum -y install virt-install yum -y install libguestfs-tools systemctl restart libvirtd 3. create sriov vfs echo 1 > /sys/bus/pci/devices/0000:5e:00.0/sriov_numvfs echo 1 > /sys/bus/pci/devices/0000:5e:00.1/sriov_numvfs 4. create VM and attach VFs to guest virsh create g1.xml virsh attach-device g1 vf1.xml virsh attach-device g1 vf2.xml * please see the attachment for the files 5. install required packages in guest yum -y install http://download-node-02.eng.bos.redhat.com/brewroot/packages/dpdk/18.11.2/1.el7/x86_64/dpdk-18.11.2-1.el7.x86_64.rpm yum -y install http://download-node-02.eng.bos.redhat.com/brewroot/packages/dpdk/18.11.2/1.el7/x86_64/dpdk-tools-18.11.2-1.el7.x86_64.rpm yum -y install libibverbs 6. set bootoptions in guest as below and reboot guest if anything changed [root@localhost ~]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-1118.el7.x86_64 root=/dev/mapper/rhel-root ro rhgb intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=2 console=ttyS0,115200 crashkernel=auto spectre_v2=retpoline rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS0 [root@localhost ~]# history 7. run testpmd testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -w 0000:03:00.0 -w 0000:04:00.0 -- --burst=64 -i --forward-mode=mac --auto-start --nb-cores=2 --rxq=1 --txq=1 --rxd=2048 --txd=2048 --disable-rss Thanks Qijun