Bug 1572840 - Broadcom card: pvp sriov case does not work well with VIOMMU inside guest [NEEDINFO]
Summary: Broadcom card: pvp sriov case does not work well with VIOMMU inside guest
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: dpdk
Version: 7.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Davide Caratti
QA Contact: liting
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-28 03:41 UTC by liting
Modified: 2020-09-02 14:31 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-02 14:31:10 UTC
Target Upstream Version:
anantha.subramanyam: needinfo? (ajit.khaparde)


Attachments (Terms of Use)

Description liting 2018-04-28 03:41:15 UTC
Description of problem:
Boardcom card: pvp sriov case does not work well with VIOMMU inside guest

Version-Release number of selected component (if applicable):
[root@netqe22 ~]# rpm -qa|grep dpdk
dpdk-17.11-10.el7fdb.x86_64
dpdk-tools-17.11-10.el7fdb.x86_64


How reproducible:


Steps to Reproduce:
1. Run vsperf pvp sriov case 
2. Inside guest, enable viommu mode, then used testpmd forward the packet
3. Used xena send rfc2544 traffic


Actual results:
The throughput got 0 mpps.
Inside guest,the PMD in the guest does not work well with VIOMMU mode 
[root@localhost bin]# [DEBUG]  2018-04-27 15:02:38,029 : (qemu_pci_passthrough) - vnf_0_cmd : ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 -- --burst=64 -i --txqflags=0xf00 --rxd=2048 --txd=2048 --nb-cores=2 --rxs
-vlan --disable-rss-rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1--disable-hw
EAL: Detected 3 lcore(s)
EAL: Some devices want iova as va but pa will be used because.. EAL: IOMMU does not support IOVA as VA
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:06.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL:   using IOMMU type 1 (Type 1)
PMD: eth_ixgbevf_dev_init(): VF Initialization Failure: -15
EAL: Releasing pci mapped resource for 0000:00:06.0
EAL: Calling pci_unmap_resource for 0000:00:06.0 at 0x7f41c0000000
EAL: Calling pci_unmap_resource for 0000:00:06.0 at 0x7f41c0004000
EAL: Requested device 0000:00:06.0 cannot be used
EAL: PCI device 0000:00:07.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 8086:10ed net_ixgbe_vf
EAL:   using IOMMU type 1 (Type 1)
PMD: eth_ixgbevf_dev_init(): VF Initialization Failure: -15
EAL: Releasing pci mapped resource for 0000:00:07.0
EAL: Calling pci_unmap_resource for 0000:00:07.0 at 0x7f41c0008000
EAL: Calling pci_unmap_resource for 0000:00:07.0 at 0x7f41c000c000
EAL: Requested device 0000:00:07.0 cannot be used
EAL: No probed ethernet devices

Expected results:
The sriov case run successfully when enable viommu inside guest.

Additional info:

Comment 3 Christian Trautman 2018-07-18 17:08:21 UTC
Hi Li Ting,

Can you please include the qemu command line and ovs commands to set this up?

Thanks,

Comment 4 liting 2018-07-20 09:37:58 UTC
In the above testing, the detail steps as following.
Run enable viommu inside guest case steps:
in host:
(1) create vf
echo 2 > /sys/devices/pci0000:80/0000:80:02.0/0000:82:00.0/sriov_numvfs
echo 3> /sys/devices/pci0000:80/0000:80:02.0/0000:82:00.1/sriov_numvfs
(2) bind the vf to dpdk
/usr/share/dpdk/usertools/dpdk-devbind.py --bind=vfio-pci 0000:82:00.3 0000:82:01.4
(3) start guest
/usr/libexec/qemu-kvm -m 8192 -smp 3 -cpu host,migratable=off -drive if=ide,file=rhel7.5-vsperf-1Q-viommu.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -M q35,kernel-irqchip=split -device intel-iommu,device-iotlb=on,intremap,caching-mode=true -device pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2 -device pcie-root-port,id=root.3,slot=3 -device pcie-root-port,id=root.4,slot=4 -device vfio-pci,host=0000:82:00.3 -device vfio-pci,host=0000:82:01.4

inside guest:
sysctl vm.nr_hugepages=1
rpm -ivh dpdk-17.11-9.el7fdb.x86_64
rpm -ivh dpdk-tools-17.11-9.el7fdb.x86_64
modprobe vfio
modprobe vfio-pci
/usr/bin/testpmd -l 0,1,2 -n 4 --socket-mem 1024 -- --burst=64 -i --txqflags=0xf00 --rxd=2048 --txd=2048 --nb-cores=2 --rxs
-vlan --disable-rss-rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1--disable-hw

Comment 5 Davide Caratti 2018-08-31 13:34:30 UTC
(un-checking 'Private' flag on comment #4, to share the reproducer steps).

hi Ajit, can you help us with this bugfix?

thank you in advance!
-- 
davide

Comment 6 Ajit Khaparde 2018-09-01 20:13:41 UTC
Davide,
Unless I am missing something in the log,
This does not look like Broadcom driver.
net_ixgbe_vf is Intel.

Thanks
Ajit

Comment 7 liting 2018-09-13 09:59:23 UTC
Sorry, the correct testpmd log of bnxt_en as following.
[root@localhost bin]# [DEBUG]  2018-09-13 04:46:23,126 : (qemu_pci_passthrough) - vnf_0_cmd : ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 -- --burst=64 -i --txqflags=0xf00 --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1--disable-hw-vlan --disable-rss
./testpmd -l 0,1,2 -n 4 --socket-mem 1024 -- --burst=64 -i 
 --txqflags=0xf00 --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1--disable-hw 
-vlan --disable-rss

EAL: Detected 3 lcore(s)

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: PCI device 0000:00:06.0 on NUMA socket -1

EAL:   Invalid NUMA socket, default to 0

EAL:   probe driver: 14e4:16dc net_bnxt

EAL:   using IOMMU type 1 (Type 1)

PMD: Broadcom Cumulus driver bnxt



PMD: Error sending msg 0x0000

PMD: bnxt_hwrm_ver_get failed rc:-1

EAL: Releasing pci mapped resource for 0000:00:06.0

EAL: Calling pci_unmap_resource for 0000:00:06.0 at 0x7fe4c0000000

EAL: Calling pci_unmap_resource for 0000:00:06.0 at 0x7fe4c0004000

EAL: Calling pci_unmap_resource for 0000:00:06.0 at 0x7fe4c0104000

EAL: Requested device 0000:00:06.0 cannot be used

EAL: PCI device 0000:00:07.0 on NUMA socket -1

EAL:   Invalid NUMA socket, default to 0

EAL:   probe driver: 14e4:16dc net_bnxt

EAL:   using IOMMU type 1 (Type 1)

PMD: Error sending msg 0x0000

PMD: bnxt_hwrm_ver_get failed rc:-1

EAL: Releasing pci mapped resource for 0000:00:07.0

EAL: Calling pci_unmap_resource for 0000:00:07.0 at 0x7fe4c0108000

EAL: Calling pci_unmap_resource for 0000:00:07.0 at 0x7fe4c010c000

EAL: Calling pci_unmap_resource for 0000:00:07.0 at 0x7fe4c020c000

EAL: Requested device 0000:00:07.0 cannot be used

EAL: No probed ethernet devices

Interactive-mode selected

USER1: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0

Done

Comment 8 svegesna 2018-09-18 09:41:37 UTC
Getting a core dump while executing the qemu-kvm command, please help us executing the command

[root@dhcp-10-123-158-137 RHEL-7.6 Server.x86_64]# /usr/libexec/qemu-kvm -m 8192 -smp 3 -cpu host,migratable=off -drive if=ide,file=/root/Ubuntu1604.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -M pc -device intel-iommu,device-iotlb=on,intremap,caching-mode=true -device pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2 -device pcie-root-port,id=root.3,slot=3 -device pcie-root-port,id=root.4,slot=4 -device vfio-pci,host=0000:86:02.0 -device vfio-pci,host=0000:86:02.1
**
ERROR:qom/object.c:409:object_new_with_type: assertion failed: (type != NULL)
Aborted (core dumped)

Comment 9 Davide Caratti 2019-06-14 08:48:10 UTC
(In reply to svegesna from comment #8)
> Getting a core dump while executing the qemu-kvm command, please help us
> executing the command
[...]

> 
> EAL:   probe driver: 14e4:16dc net_bnxt
> EAL:   using IOMMU type 1 (Type 1)
> PMD: Broadcom Cumulus driver bnxt
> PMD: Error sending msg 0x0000
> PMD: bnxt_hwrm_ver_get failed rc:-1

hello,

this was fixed with the IOVA_AS_VA patch (https://mails.dpdk.org/archives/dev/2019-February/124663.html, bz1671422). Can you please retest it on latest FDP rpm, and see if the PMD segfault is still there?

thank you in advance,
-- 
davide

Comment 10 Marcelo Ricardo Leitner 2019-07-03 15:11:09 UTC
Mass-moving bugs RHEL <= 7.6.0 to 7.7.0.
As we are past RFE deadline for 7.7.0 and we should have no new features on 7.8.0, please evaluate if it's still wanted on RHEL7 and contact PM for exception. You may also move it to RHEL8 if that's wanted. Thanks!

Comment 12 liting 2020-05-08 10:55:48 UTC
Hi Davide,

I run 64bytes 1q 2pmd got 16mpps. host use dpdk-18.11.5-1.el7_8.x86_64.rpm, guest use dpdk-18.11.2-1.el7_6.x86_64.rpm. Host and guest are rhel7.7.
Following is the detail info.

[root@netqe22 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@netqe22 ~]# taskset -c 3,5,29 /usr/libexec/qemu-kvm -m 8192 -smp 3 -cpu host,migratable=off -drive if=ide,file=rhel7.7-vsperf-1Q-viommu.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -M q35,kernel-irqchip=split -device intel-iommu,device-iotlb=on,intremap,caching-mode=true -device pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2 -device pcie-root-port,id=root.3,slot=3 -device pcie-root-port,id=root.4,slot=4 -device vfio-pci,bus=root.2,host=0000:82:02.1 -device vfio-pci,bus=root.3,host=0000:82:04.1
Inside guest, start testpmd with following error
[root@localhost ~]# rpm -ivh /root/dpdkrpms/dpdk*.rpm 
rpm -ivh /root/dpdkrpms/dpdk*.rpm 
Preparing...                          #[DEBUG]  2020-05-08 05:19:59,832 : (qemu_pci_passthrough) - vnf_0_cmd : cat /proc/cmdline
cat /proc/cmdline################################# [100%]
Updating / installing...
   1:dpdk-18.11.2-1.el7_6             
################################# [ 50%]
   2:dpdk-tools-18.11.2-1.el7_6       ################################# [100%]


[root@localhost ~]# modprobe vfio
modprobe vfio
[root@localhost ~]# modprobe vfio-pci
modprobe vfio-pci
[root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci 02:00.0 03:00.0
[root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py --status
/usr/share/dpdk/usertools/dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:02:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci unused=bnxt_en
0000:03:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci unused=bnxt_en


[root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem s
--burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1 --disable-rss
EAL: Detected 3 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 14e4:16dc net_bnxt
EAL:   using IOMMU type 1 (Type 1)
bnxt_dev_init(): Broadcom NetXtreme driver bnxt

bnxt_dev_init(): Memzone physical address same as virtual using rte_mem_virt2iova()
bnxt_dev_init(): Memzone physical address same as virtual.
bnxt_dev_init(): Using rte_mem_virt2iova()
bnxt_hwrm_ver_get(): 1.8.1:20.6.143
bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
bnxt_dev_init(): bnxt found at mem fd500000, node addr 0x940000000M
bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL:   Invalid NUMA socket, default to 0
EAL:   probe driver: 14e4:16dc net_bnxt
bnxt_dev_init(): Memzone physical address same as virtual using rte_mem_virt2iova()
bnxt_dev_init(): Memzone physical address same as virtual.
bnxt_dev_init(): Using rte_mem_virt2iova()
bnxt_hwrm_ver_get(): 1.8.1:20.6.143
bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
bnxt_dev_init(): bnxt found at mem fd300000, node addr 0x940108000M
bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
Interactive-mode selected
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
bnxt_alloc_rings(): Memzone physical address same as virtual.
bnxt_alloc_rings(): Using rte_mem_virt2iova()
bnxt_alloc_rings(): Memzone physical address same as virtual.
bnxt_alloc_rings(): Using rte_mem_virt2iova()
bnxt_print_link_info(): Port 0 Link Down

Port 0: link state change event
bnxt_print_link_info(): Port 0 Link Up - speed 10000 Mbps - full-duplex
Port 0: FA:42:D3:A6:B9:F3
Configuring Port 1 (socket 0)
bnxt_alloc_rings(): Memzone physical address same as virtual.
bnxt_alloc_rings(): Using rte_mem_virt2iova()
bnxt_alloc_rings(): Memzone physical address same as virtual.
bnxt_alloc_rings(): Using rte_mem_virt2iova()
bnxt_print_link_info(): Port 1 Link Down

Port 1: link state change event
bnxt_print_link_info(): Port 1 Link Up - speed 10000 Mbps - full-duplex
Port 1: 86:CB:E1:F8:C4:32
Checking link statuses...
Done

But I still run this case failed on rhel8.1. host use dpdk-19.11-1.el8.x86_64.rpm. guest use dpdk-18.11.2-4.el8.x86_64.rpm. 
The testpmd start with following error inside guest.
[root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1 --disable-rss
./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- -
--burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1 --disable-rss

EAL: Detected 3 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: PCI device 0000:02:00.0 on NUMA socket -1

EAL:   Invalid NUMA socket, default to 0

EAL:   probe driver: 14e4:16dc net_bnxt

EAL:   using IOMMU type 1 (Type 1)

bnxt_dev_init(): Broadcom NetXtreme driver bnxt



bnxt_dev_init(): Memzone physical address same as virtual using rte_mem_virt2iova()

bnxt_dev_init(): Memzone physical address same as virtual.

bnxt_dev_init(): Using rte_mem_virt2iova()

bnxt_hwrm_ver_get(): 1.8.1:20.6.143

bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2

bnxt_dev_init(): Invalid MAC addr 00:00:00:00:00:00

Segmentation fault (core dumped)


thanks,
Li Ting

Comment 13 Davide Caratti 2020-05-08 13:46:23 UTC
(In reply to liting from comment #12)
> Hi Davide,
> 
> I run 64bytes 1q 2pmd got 16mpps. host use dpdk-18.11.5-1.el7_8.x86_64.rpm,
> guest use dpdk-18.11.2-1.el7_6.x86_64.rpm. Host and guest are rhel7.7.
> Following is the detail info.
> 
> [root@netqe22 ~]# cat /etc/redhat-release 
> Red Hat Enterprise Linux Server release 7.7 (Maipo)
> 
> [root@netqe22 ~]# taskset -c 3,5,29 /usr/libexec/qemu-kvm -m 8192 -smp 3
> -cpu host,migratable=off -drive if=ide,file=rhel7.7-vsperf-1Q-viommu.qcow2
> -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object
> memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa
> node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot
> -net none -no-reboot -M q35,kernel-irqchip=split -device
> intel-iommu,device-iotlb=on,intremap,caching-mode=true -device
> pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2
> -device pcie-root-port,id=root.3,slot=3 -device
> pcie-root-port,id=root.4,slot=4 -device
> vfio-pci,bus=root.2,host=0000:82:02.1 -device
> vfio-pci,bus=root.3,host=0000:82:04.1
> Inside guest, start testpmd with following error
> [root@localhost ~]# rpm -ivh /root/dpdkrpms/dpdk*.rpm 
> rpm -ivh /root/dpdkrpms/dpdk*.rpm 
> Preparing...                          #[DEBUG]  2020-05-08 05:19:59,832 :
> (qemu_pci_passthrough) - vnf_0_cmd : cat /proc/cmdline
> cat /proc/cmdline################################# [100%]
> Updating / installing...
>    1:dpdk-18.11.2-1.el7_6             
> ################################# [ 50%]
>    2:dpdk-tools-18.11.2-1.el7_6       #################################
> [100%]
> 
> 
> [root@localhost ~]# modprobe vfio
> modprobe vfio
> [root@localhost ~]# modprobe vfio-pci
> modprobe vfio-pci
> [root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci
> 02:00.0 03:00.0
> [root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py --status
> /usr/share/dpdk/usertools/dpdk-devbind.py --status
> 
> Network devices using DPDK-compatible driver
> ============================================
> 0000:02:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci
> unused=bnxt_en
> 0000:03:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci
> unused=bnxt_en
> 
> 
> [root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem
> s
> --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> --disable-rss
> EAL: Detected 3 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: PCI device 0000:02:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 14e4:16dc net_bnxt
> EAL:   using IOMMU type 1 (Type 1)
> bnxt_dev_init(): Broadcom NetXtreme driver bnxt
> 
> bnxt_dev_init(): Memzone physical address same as virtual using
> rte_mem_virt2iova()
> bnxt_dev_init(): Memzone physical address same as virtual.
> bnxt_dev_init(): Using rte_mem_virt2iova()
> bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> bnxt_dev_init(): bnxt found at mem fd500000, node addr 0x940000000M
> bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
> bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
> EAL: PCI device 0000:03:00.0 on NUMA socket -1
> EAL:   Invalid NUMA socket, default to 0
> EAL:   probe driver: 14e4:16dc net_bnxt
> bnxt_dev_init(): Memzone physical address same as virtual using
> rte_mem_virt2iova()
> bnxt_dev_init(): Memzone physical address same as virtual.
> bnxt_dev_init(): Using rte_mem_virt2iova()
> bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> bnxt_dev_init(): bnxt found at mem fd300000, node addr 0x940108000M
> bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
> bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
> Interactive-mode selected
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
> socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> Configuring Port 0 (socket 0)
> bnxt_alloc_rings(): Memzone physical address same as virtual.
> bnxt_alloc_rings(): Using rte_mem_virt2iova()
> bnxt_alloc_rings(): Memzone physical address same as virtual.
> bnxt_alloc_rings(): Using rte_mem_virt2iova()
> bnxt_print_link_info(): Port 0 Link Down
> 
> Port 0: link state change event
> bnxt_print_link_info(): Port 0 Link Up - speed 10000 Mbps - full-duplex
> Port 0: FA:42:D3:A6:B9:F3
> Configuring Port 1 (socket 0)
> bnxt_alloc_rings(): Memzone physical address same as virtual.
> bnxt_alloc_rings(): Using rte_mem_virt2iova()
> bnxt_alloc_rings(): Memzone physical address same as virtual.
> bnxt_alloc_rings(): Using rte_mem_virt2iova()
> bnxt_print_link_info(): Port 1 Link Down
> 
> Port 1: link state change event
> bnxt_print_link_info(): Port 1 Link Up - speed 10000 Mbps - full-duplex
> Port 1: 86:CB:E1:F8:C4:32
> Checking link statuses...
> Done
> 
> But I still run this case failed on rhel8.1. host use
> dpdk-19.11-1.el8.x86_64.rpm. guest use dpdk-18.11.2-4.el8.x86_64.rpm. 
> The testpmd start with following error inside guest.
> [root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem
> -- --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> --disable-rss
> ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- -
> --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> --disable-rss
> 
> EAL: Detected 3 lcore(s)
> 
> EAL: Detected 1 NUMA nodes
> 
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> 
> EAL: Probing VFIO support...
> 
> EAL: VFIO support initialized
> 
> EAL: PCI device 0000:02:00.0 on NUMA socket -1
> 
> EAL:   Invalid NUMA socket, default to 0
> 
> EAL:   probe driver: 14e4:16dc net_bnxt
> 
> EAL:   using IOMMU type 1 (Type 1)
> 
> bnxt_dev_init(): Broadcom NetXtreme driver bnxt
> 
> 
> 
> bnxt_dev_init(): Memzone physical address same as virtual using
> rte_mem_virt2iova()
> 
> bnxt_dev_init(): Memzone physical address same as virtual.
> 
> bnxt_dev_init(): Using rte_mem_virt2iova()
> 
> bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> 
> bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> 
> bnxt_dev_init(): Invalid MAC addr 00:00:00:00:00:00
> 
> Segmentation fault (core dumped)
> 
> 
> thanks,
> Li Ting

hello Li Ting,

apparently bnxt_dev_init() for the VF is failing badly. Is this happening also
using  dpdk-18.11.2-3.el8.x86_64.rpm ?

thanks!
-- 
davide

Comment 14 liting 2020-05-11 06:56:07 UTC
(In reply to Davide Caratti from comment #13)
> (In reply to liting from comment #12)
> > Hi Davide,
> > 
> > I run 64bytes 1q 2pmd got 16mpps. host use dpdk-18.11.5-1.el7_8.x86_64.rpm,
> > guest use dpdk-18.11.2-1.el7_6.x86_64.rpm. Host and guest are rhel7.7.
> > Following is the detail info.
> > 
> > [root@netqe22 ~]# cat /etc/redhat-release 
> > Red Hat Enterprise Linux Server release 7.7 (Maipo)
> > 
> > [root@netqe22 ~]# taskset -c 3,5,29 /usr/libexec/qemu-kvm -m 8192 -smp 3
> > -cpu host,migratable=off -drive if=ide,file=rhel7.7-vsperf-1Q-viommu.qcow2
> > -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object
> > memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa
> > node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot
> > -net none -no-reboot -M q35,kernel-irqchip=split -device
> > intel-iommu,device-iotlb=on,intremap,caching-mode=true -device
> > pcie-root-port,id=root.1,slot=1 -device pcie-root-port,id=root.2,slot=2
> > -device pcie-root-port,id=root.3,slot=3 -device
> > pcie-root-port,id=root.4,slot=4 -device
> > vfio-pci,bus=root.2,host=0000:82:02.1 -device
> > vfio-pci,bus=root.3,host=0000:82:04.1
> > Inside guest, start testpmd with following error
> > [root@localhost ~]# rpm -ivh /root/dpdkrpms/dpdk*.rpm 
> > rpm -ivh /root/dpdkrpms/dpdk*.rpm 
> > Preparing...                          #[DEBUG]  2020-05-08 05:19:59,832 :
> > (qemu_pci_passthrough) - vnf_0_cmd : cat /proc/cmdline
> > cat /proc/cmdline################################# [100%]
> > Updating / installing...
> >    1:dpdk-18.11.2-1.el7_6             
> > ################################# [ 50%]
> >    2:dpdk-tools-18.11.2-1.el7_6       #################################
> > [100%]
> > 
> > 
> > [root@localhost ~]# modprobe vfio
> > modprobe vfio
> > [root@localhost ~]# modprobe vfio-pci
> > modprobe vfio-pci
> > [root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci
> > 02:00.0 03:00.0
> > [root@localhost ~]# /usr/share/dpdk/usertools/dpdk-devbind.py --status
> > /usr/share/dpdk/usertools/dpdk-devbind.py --status
> > 
> > Network devices using DPDK-compatible driver
> > ============================================
> > 0000:02:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci
> > unused=bnxt_en
> > 0000:03:00.0 'NetXtreme-E Ethernet Virtual Function 16dc' drv=vfio-pci
> > unused=bnxt_en
> > 
> > 
> > [root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem
> > s
> > --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> > --disable-rss
> > EAL: Detected 3 lcore(s)
> > EAL: Detected 1 NUMA nodes
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > EAL: Probing VFIO support...
> > EAL: VFIO support initialized
> > EAL: PCI device 0000:02:00.0 on NUMA socket -1
> > EAL:   Invalid NUMA socket, default to 0
> > EAL:   probe driver: 14e4:16dc net_bnxt
> > EAL:   using IOMMU type 1 (Type 1)
> > bnxt_dev_init(): Broadcom NetXtreme driver bnxt
> > 
> > bnxt_dev_init(): Memzone physical address same as virtual using
> > rte_mem_virt2iova()
> > bnxt_dev_init(): Memzone physical address same as virtual.
> > bnxt_dev_init(): Using rte_mem_virt2iova()
> > bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> > bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> > bnxt_dev_init(): bnxt found at mem fd500000, node addr 0x940000000M
> > bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
> > bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
> > EAL: PCI device 0000:03:00.0 on NUMA socket -1
> > EAL:   Invalid NUMA socket, default to 0
> > EAL:   probe driver: 14e4:16dc net_bnxt
> > bnxt_dev_init(): Memzone physical address same as virtual using
> > rte_mem_virt2iova()
> > bnxt_dev_init(): Memzone physical address same as virtual.
> > bnxt_dev_init(): Using rte_mem_virt2iova()
> > bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> > bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> > bnxt_dev_init(): bnxt found at mem fd300000, node addr 0x940108000M
> > bnxt_alloc_vnic_attributes(): Memzone physical address same as virtual.
> > bnxt_alloc_vnic_attributes(): Using rte_mem_virt2iova()
> > Interactive-mode selected
> > testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176,
> > socket=0
> > testpmd: preferred mempool ops selected: ring_mp_mc
> > Configuring Port 0 (socket 0)
> > bnxt_alloc_rings(): Memzone physical address same as virtual.
> > bnxt_alloc_rings(): Using rte_mem_virt2iova()
> > bnxt_alloc_rings(): Memzone physical address same as virtual.
> > bnxt_alloc_rings(): Using rte_mem_virt2iova()
> > bnxt_print_link_info(): Port 0 Link Down
> > 
> > Port 0: link state change event
> > bnxt_print_link_info(): Port 0 Link Up - speed 10000 Mbps - full-duplex
> > Port 0: FA:42:D3:A6:B9:F3
> > Configuring Port 1 (socket 0)
> > bnxt_alloc_rings(): Memzone physical address same as virtual.
> > bnxt_alloc_rings(): Using rte_mem_virt2iova()
> > bnxt_alloc_rings(): Memzone physical address same as virtual.
> > bnxt_alloc_rings(): Using rte_mem_virt2iova()
> > bnxt_print_link_info(): Port 1 Link Down
> > 
> > Port 1: link state change event
> > bnxt_print_link_info(): Port 1 Link Up - speed 10000 Mbps - full-duplex
> > Port 1: 86:CB:E1:F8:C4:32
> > Checking link statuses...
> > Done
> > 
> > But I still run this case failed on rhel8.1. host use
> > dpdk-19.11-1.el8.x86_64.rpm. guest use dpdk-18.11.2-4.el8.x86_64.rpm. 
> > The testpmd start with following error inside guest.
> > [root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem
> > -- --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> > --disable-rss
> > ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- -
> > --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1
> > --disable-rss
> > 
> > EAL: Detected 3 lcore(s)
> > 
> > EAL: Detected 1 NUMA nodes
> > 
> > EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> > 
> > EAL: Probing VFIO support...
> > 
> > EAL: VFIO support initialized
> > 
> > EAL: PCI device 0000:02:00.0 on NUMA socket -1
> > 
> > EAL:   Invalid NUMA socket, default to 0
> > 
> > EAL:   probe driver: 14e4:16dc net_bnxt
> > 
> > EAL:   using IOMMU type 1 (Type 1)
> > 
> > bnxt_dev_init(): Broadcom NetXtreme driver bnxt
> > 
> > 
> > 
> > bnxt_dev_init(): Memzone physical address same as virtual using
> > rte_mem_virt2iova()
> > 
> > bnxt_dev_init(): Memzone physical address same as virtual.
> > 
> > bnxt_dev_init(): Using rte_mem_virt2iova()
> > 
> > bnxt_hwrm_ver_get(): 1.8.1:20.6.143
> > 
> > bnxt_hwrm_ver_get(): Driver HWRM version: 1.9.2
> > 
> > bnxt_dev_init(): Invalid MAC addr 00:00:00:00:00:00
> > 
> > Segmentation fault (core dumped)
> > 
> > 
> > thanks,
> > Li Ting
> 
> hello Li Ting,
> 
> apparently bnxt_dev_init() for the VF is failing badly. Is this happening
> also
> using  dpdk-18.11.2-3.el8.x86_64.rpm ?
> 
> thanks!
> -- 
> davide

Hi Davide,

I change to use dpdk-18.11.2-3.el8.x86_64.rpm inside guest, The same issue is still happening. In addition, I run the same case on rhel8.2, both host and guest use dpdk-19.11-4.el8fdb.1.x86_64.rpm, it run failed too.
Following is the testpmd info when using dpdk-19.11-4.el8fdb.1.x86_64.rpm on rhel8.2 guest
[root@localhost bin]# ./testpmd -l 0,1,2 -n 4 --socket-mem 1024 --legacy-mem -- --burst=64 -i --rxd=2048 --txd=2048 --nb-cores=2 --rxq=1 --txq=1 --disable-rss

EAL: Detected 3 lcore(s)

EAL: Detected 1 NUMA nodes

EAL: Multi-process socket /var/run/dpdk/rte/mp_socket

EAL: Selected IOVA mode 'VA'

EAL: Probing VFIO support...

EAL: VFIO support initialized

EAL: PCI device 0000:02:00.0 on NUMA socket -1

EAL:   Invalid NUMA socket, default to 0

EAL:   probe driver: 14e4:16dc net_bnxt

EAL:   using IOMMU type 1 (Type 1)

bnxt_hwrm_set_mac(): error 3:0:00000000:0000

EAL: Releasing pci mapped resource for 0000:02:00.0

EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x940000000

EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x940004000

EAL: Calling pci_unmap_resource for 0000:02:00.0 at 0x940104000

EAL: Requested device 0000:02:00.0 cannot be used

EAL: PCI device 0000:03:00.0 on NUMA socket -1

EAL:   Invalid NUMA socket, default to 0

EAL:   probe driver: 14e4:16dc net_bnxt

EAL:   using IOMMU type 1 (Type 1)

bnxt_hwrm_set_mac(): error 3:0:00000000:0000

EAL: Releasing pci mapped resource for 0000:03:00.0

EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x940108000

EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x94010c000

EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x94020c000

EAL: Requested device 0000:03:00.0 cannot be used

testpmd: No probed ethernet devices

Interactive-mode selected

Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0

EAL: Error - exiting with code: 1

  Cause: rxq 1 invalid - must be >= 0 && <= 0


thanks,
Li Ting

Comment 15 Davide Caratti 2020-05-28 09:09:51 UTC
unset 'Private' to comment #12 , comment #13 , comment #14  so that Broadcom CCX team can look at them.
Also, moving the 'needinfo' on Anantha (who can help better with root cause analysis).

Comment 16 Anantha Subramanyam 2020-06-03 15:09:54 UTC
Ajit, as discussed offline passing this onto you, thanks

Comment 17 Anantha Subramanyam 2020-09-01 23:27:21 UTC
Per our internal discussion Broadcom is ok to close this old BZ.

Comment 18 Gary Case 2020-09-02 14:31:10 UTC
Thanks, Ananth. I'll close the bug.


Note You need to log in before you can comment on or make changes to this bug.