Description of problem: bnxt_en: testpmd as switch case run failed with dpdk-21.11 Version-Release number of selected component (if applicable): [root@netqe22 perf]# rpm -qa|grep dpdk dpdk-21.11-1.el8.x86_64 [root@netqe22 perf]# uname -r 4.18.0-365.el8.x86_64 [root@netqe22 perf]# ethtool -i enp130s0f0np0 driver: bnxt_en version: 4.18.0-365.el8.x86_64 firmware-version: 20.6.143.0/pkg 20.06.04.06 expansion-rom-version: bus-info: 0000:82:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no How reproducible: Steps to Reproduce: Run testpmd as switch case 1. bind two port to dpdk [root@netqe22 perf]# driverctl -v set-override 0000:82:00.0 vfio-pci [root@netqe22 perf]# driverctl -v set-override 0000:82:00.1 vfio-pci 2. Run testpmd command as following [root@netqe22 perf]# /usr/bin/dpdk-testpmd -l 47,23,45 -n 4 --socket-mem 1024,1024 --vdev net_vhost0,iface=/tmp/vhost0,client=1,iommu-support=1,queues=1 --vdev net_vhost1,iface=/tmp/vhost1,client=1,iommu-support=1,queues=1 -- -i --nb-cores=2 --txq=1 --rxq=1 --forward-mode=io EAL: Detected CPU lcores: 48 EAL: Detected NUMA nodes: 2 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 2048 kB hugepages reported EAL: VFIO support initialized EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_bnxt (14e4:16d7) device: 0000:82:00.0 (socket 1) ethdev initialisation failed EAL: Releasing PCI mapped resource for 0000:82:00.0 EAL: Calling pci_unmap_resource for 0000:82:00.0 at 0x2200000000 EAL: Calling pci_unmap_resource for 0000:82:00.0 at 0x2200010000 EAL: Calling pci_unmap_resource for 0000:82:00.0 at 0x2200110000 EAL: Requested device 0000:82:00.0 cannot be used EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_bnxt (14e4:16d7) device: 0000:82:00.1 (socket 1) ethdev initialisation failed EAL: Releasing PCI mapped resource for 0000:82:00.1 EAL: Calling pci_unmap_resource for 0000:82:00.1 at 0x2200112000 EAL: Calling pci_unmap_resource for 0000:82:00.1 at 0x2200122000 EAL: Calling pci_unmap_resource for 0000:82:00.1 at 0x2200222000 EAL: Requested device 0000:82:00.1 cannot be used TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected Set io packet forwarding mode testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) VHOST_CONFIG: vhost-user client: socket created, fd: 27 VHOST_CONFIG: new device, handle is 0, path is /tmp/vhost0 Port 0: 56:48:4F:53:54:00 Configuring Port 1 (socket 1) VHOST_CONFIG: vhost-user client: socket created, fd: 31 VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: new device, handle is 1, path is /tmp/vhost1 Port 1: 56:48:4F:53:54:01 Checking link statuses... VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:33 VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:34 VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:36 VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:37 Done testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ Actual results: The testpmd should has four ports, testpmd has no two physic ports. Expected results: The dpdk-20.11-3.el8 can work well as following. [root@netqe22 perf]# /usr/bin/dpdk-testpmd -l 47,23,45 -n 4 --socket-mem 1024,1024 --vdev net_vhost0,iface=/tmp/vhost0,client=1,iommu-support=1,queues=1 --vdev net_vhost1,iface=/tmp/vhost1,client=1,iommu-support=1,queues=1 -- -i --nb-cores=2 --txq=1 --rxq=1 --forward-mode=io EAL: Detected 48 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_bnxt (14e4:16d7) device: 0000:82:00.0 (socket 1) EAL: Probe PCI driver: net_bnxt (14e4:16d7) device: 0000:82:00.1 (socket 1) EAL: No legacy callbacks, legacy socket not created Interactive-mode selected Set io packet forwarding mode testpmd: create a new mbuf pool <mb_pool_1>: n=163456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) Port 0: link state change event Port 0: 00:0A:F7:B7:09:50 Configuring Port 1 (socket 1) Port 1: link state change event Port 1: 00:0A:F7:B7:09:51 Configuring Port 2 (socket 1) VHOST_CONFIG: vhost-user client: socket created, fd: 38 VHOST_CONFIG: new device, handle is 0 Port 2: 56:48:4F:53:54:02 Configuring Port 3 (socket 1) VHOST_CONFIG: vhost-user client: socket created, fd: 41 VHOST_CONFIG: new device, handle is 1 Port 3: 56:48:4F:53:54:03 Checking link statuses... VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:43 VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:44 VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_GET_PROTOCOL_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_PROTOCOL_FEATURES VHOST_CONFIG: negotiated Vhost-user protocol features: 0xcbf VHOST_CONFIG: read message VHOST_USER_GET_QUEUE_NUM VHOST_CONFIG: read message VHOST_USER_SET_SLAVE_REQ_FD VHOST_CONFIG: read message VHOST_USER_SET_OWNER VHOST_CONFIG: read message VHOST_USER_GET_FEATURES VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:0 file:46 VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL VHOST_CONFIG: vring call idx:1 file:47 Done testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 1 RX-missed: 0 RX-bytes: 110 RX-errors: 12001282 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ ######################## NIC statistics for port 1 ######################## RX-packets: 1 RX-missed: 0 RX-bytes: 110 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ ######################## NIC statistics for port 2 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ ######################## NIC statistics for port 3 ######################## RX-packets: 0 RX-missed: 0 RX-bytes: 0 RX-errors: 0 RX-nombuf: 0 TX-packets: 0 TX-errors: 0 TX-bytes: 0 Throughput (since last show) Rx-pps: 0 Rx-bps: 0 Tx-pps: 0 Tx-bps: 0 ############################################################################ testpmd> Additional info: https://beaker.engineering.redhat.com/jobs/6335929
sriov case also run failed. testpmd command inside guest: dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start EAL: Detected CPU lcores: 3 EAL: Detected NUMA nodes: 1 EAL: Detected shared linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'VA' EAL: No available 2048 kB hugepages reported EAL: VFIO support initialized EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0) eth_virtio_pci_init(): Failed to init PCI device EAL: Requested device 0000:02:00.0 cannot be used EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:03:00.0 (socket 0) ethdev initialisation failed EAL: Releasing PCI mapped resource for 0000:03:00.0 EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1180000000 EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1180004000 EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x1180104000 EAL: Requested device 0000:03:00.0 cannot be used EAL: Using IOMMU type 1 (Type 1) EAL: Probe PCI driver: net_bnxt (14e4:16dc) device: 0000:04:00.0 (socket 0) ethdev initialisation failed EAL: Releasing PCI mapped resource for 0000:04:00.0 EAL: Calling pci_unmap_resource for 0000:04:00.0 at 0x1180108000 EAL: Calling pci_unmap_resource for 0000:04:00.0 at 0x118010c000 EAL: Calling pci_unmap_resource for 0000:04:00.0 at 0x118020c000 EAL: Requested device 0000:04:00.0 cannot be used TELEMETRY: No legacy callbacks, legacy socket not created testpmd: No probed ethernet devices Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0 EAL: Error - exiting with code: 1 Cause: rxq 1 invalid - must be >= 0 && <= 0 Interactive-mode selected And dpdk-21.11-1.el9 is also has this issue. https://beaker.engineering.redhat.com/jobs/6336885
It seems to be the same issue than bz2055531.
For dpdk-21.11.2-1.el9_1.x86_64 still has this issue. https://beaker.engineering.redhat.com/jobs/7123767 https://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2022/10/71237/7123767/12777018/151553290/bnxt_25.html