Description of problem: container start testpmd failed Version-Release number of selected component (if applicable): kernel version: 5.14.0-284.3.1.el9_2.x86_64 podman-4.4.1-3.el9.x86_64 dpdk-22.11-1.el9.x86_64 dpdk-tools-22.11-1.el9.x86_64 test nic: # ethtool -i ens4f0 driver: ice version: 5.14.0-284.3.1.el9_2.x86_64 firmware-version: 3.20 0x8000d83e 1.3146.0 expansion-rom-version: bus-info: 0000:b1:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes How reproducible:always Steps to Reproduce: 1. enable hugepage, cpu tunning, iommu on host kernel commandline [root@dell-per750-13 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-284.3.1.el9_2.x86_64 root=/dev/mapper/rhel_dell--per750--13-root ro ksdevice=bootif pci=realloc crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_dell--per750--13-swap rd.lvm.lv=rhel_dell-per750-13/root rd.lvm.lv=rhel_dell-per750-13/swap console=ttyS0,115200n81 skew_tick=1 nohz=on nohz_full=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 rcu_nocbs=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 tuned.non_isolcpus=00000003 intel_pstate=disable nosoftlockup default_hugepagesz=1G hugepagesz=1G hugepages=30 isolcpus=2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31 intel_iommu=on iommu=pt intel_idle.max_cstate=0 processor.max_cstate=0 intel_pstate=disable idle=poll rcu_nocb_poll 2. create vf from specify pf, then reload to vfio-pci driver [root@dell-per750-13 ~]# lsmod | grep vfio vfio_pci 16384 0 vfio_pci_core 69632 1 vfio_pci vfio_virqfd 16384 1 vfio_pci_core vfio_iommu_type1 49152 0 vfio 40960 2 vfio_pci_core,vfio_iommu_type1 irqbypass 16384 2 vfio_pci_core,kvm echo 1 > /sys/class/net/ens4f0/device/sriov_numvfs echo 1 > /sys/class/net/ens4f1/device/sriov_numvfs [root@dell-per750-13 ~]# driverctl -v set-override 0000:b1:11.0 vfio-pci driverctl: setting driver override for 0000:b1:11.0: vfio-pci driverctl: loading driver vfio-pci driverctl: unbinding previous driver iavf driverctl: reprobing driver for 0000:b1:11.0 driverctl: saving driver override for 0000:b1:11.0 [root@dell-per750-13 ~]# driverctl -v set-override 0000:b1:01.0 vfio-pci driverctl: setting driver override for 0000:b1:01.0: vfio-pci driverctl: loading driver vfio-pci driverctl: unbinding previous driver iavf driverctl: reprobing driver for 0000:b1:01.0 driverctl: saving driver override for 0000:b1:01.0 3. create container podman run --name=test1 -dt --privileged --cpuset-mems=1 --cpuset-cpus=5,7,9 -v /dev/vfio/vfio:/dev/vfio/vfio -v /dev/hugepages:/dev/hugepages 8d236f403e2c /bin/bash 4. install dpdk/dpdk-tools on container and start testpmd [root@dcd73075e53a /]# rpm -qa | grep dpdk dpdk-22.11-1.el9.x86_64 dpdk-tools-22.11-1.el9.x86_64 [root@dcd73075e53a /]# dpdk-testpmd --log-level="*:debug" -a 0000:b1:01.0 -a 0000:b1:11.0 -n 4 -m 1024 -- -i EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: bus.auxiliary log level changed from notice to debug EAL: lib.rcu log level changed from error to debug EAL: pmd.common.mlx5 log level changed from notice to debug EAL: pmd.common.iavf log level changed from notice to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: pmd.net.mlx5 log level changed from notice to debug EAL: pmd.net.ice.driver log level changed from notice to debug EAL: pmd.net.ice.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.23.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.23.0 EAL: Registered [vdev] bus. EAL: bus.vdev log level changed from disabled to debug EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.23.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.23.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.23.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.23.0 EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.23.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.23.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.23.0 EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.23.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.23.0 EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: pmd.net.netvsc.init log level changed from disabled to debug rte_vmbus_register(): Registered driver net_netvsc EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.23.0 EAL: pmd.net.nfp.cpp log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.23.0 EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: pmd.net.qede.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.23.0 EAL: lib.dmadev log level changed from disabled to debug EAL: Registered [vdpa] device class. EAL: lib.vhost.data log level changed from disabled to debug EAL: lib.vhost.config log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.23.0 EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) EAL: Failed to create thread for interrupt handling EAL: FATAL: Cannot init interrupt-handling thread EAL: Cannot init interrupt-handling thread EAL: Error - exiting with code: 1 Cause: Cannot init EAL: Invalid argument Actual results: Frome step4, testpmd start failed Expected results: From step4, testpmd can work as well Additional info: Host start testpmd as well [root@dell-per750-13 ~]# dpdk-testpmd --log-level="*:debug" -a 0000:b1:01.0 -a 0000:b1:11.0 -n 4 -m 1024 -- -i EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: bus.auxiliary log level changed from notice to debug EAL: lib.rcu log level changed from error to debug EAL: pmd.common.mlx5 log level changed from notice to debug EAL: pmd.common.iavf log level changed from notice to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: pmd.net.mlx5 log level changed from notice to debug EAL: pmd.net.ice.driver log level changed from notice to debug EAL: pmd.net.ice.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.23.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.23.0 EAL: Registered [vdev] bus. EAL: bus.vdev log level changed from disabled to debug EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.23.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.23.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.23.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.23.0 EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.23.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.23.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.23.0 EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.23.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.23.0 EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: pmd.net.netvsc.init log level changed from disabled to debug rte_vmbus_register(): Registered driver net_netvsc EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.23.0 EAL: pmd.net.nfp.cpp log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.23.0 EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: pmd.net.qede.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.23.0 EAL: lib.dmadev log level changed from disabled to debug EAL: Registered [vdpa] device class. EAL: lib.vhost.data log level changed from disabled to debug EAL: lib.vhost.config log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.23.0 EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Bus auxiliary wants IOVA as 'DC' EAL: PCI driver net_ice_dcf for device 0000:b1:01.0 wants IOVA as 'DC' EAL: PCI driver net_iavf for device 0000:b1:01.0 wants IOVA as 'DC' EAL: PCI driver net_ice_dcf for device 0000:b1:11.0 wants IOVA as 'DC' EAL: PCI driver net_iavf for device 0000:b1:11.0 wants IOVA as 'DC' EAL: Bus pci wants IOVA as 'DC' EAL: Bus vdev wants IOVA as 'DC' EAL: Buses did not request a specific IOVA mode. EAL: IOMMU is available, selecting IOVA as VA mode. EAL: Selected IOVA mode 'VA' EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is supported EAL: IOMMU type 7 (sPAPR) is not supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x2e000 bytes EAL: Virtual area found at 0x100007000 (size = 0x2e000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 524288 EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 EAL: Detected memory type: socket_id:1 hugepage_sz:1073741824 EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x100035000 (size = 0x1000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x140000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x140000000, size 800000000 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x940000000 (size = 0x1000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x980000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x980000000, size 800000000 EAL: Creating 2 segment lists: n_segs:32 socket_id:1 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x1180000000 (size = 0x1000) EAL: Memseg list allocated at socket 1, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x11c0000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x11c0000000, size 800000000 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x19c0000000 (size = 0x1000) EAL: Memseg list allocated at socket 1, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x1a00000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x1a00000000, size 800000000 EAL: Allocating 1 pages of size 1024M on socket 0 EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 0 EAL: Restoring previous memory policy: 0 EAL: Allocating 1 pages of size 1024M on socket 1 EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 1 EAL: Restoring previous memory policy: 0 EAL: Added 1024M to heap on socket 0 EAL: Added 1024M to heap on socket 1 EAL: TSC frequency is ~2900000 KHz EAL: Main lcore 0 is ready (tid=7fa5a8e74900;cpuset=[0]) EAL: lcore 1 is ready (tid=7fa5a7305640;cpuset=[1]) EAL: PCI device 0000:b1:01.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_ice_dcf EAL: Using IOMMU type 1 (Type 1) EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered EAL: Installed memory event callback for VFIO EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x2200000000 EAL: PCI memory mapped at 0x2200020000 EAL: Probe PCI driver: net_ice_dcf (8086:1889) device: 0000:b1:01.0 (socket 1) EAL: Mem event callback 'vfio_mem_event_clb:(nil)' unregistered EAL: Releasing PCI mapped resource for 0000:b1:01.0 EAL: Calling pci_unmap_resource for 0000:b1:01.0 at 0x2200000000 EAL: PCI memory unmapped at 0x2200000000 EAL: Calling pci_unmap_resource for 0000:b1:01.0 at 0x2200020000 EAL: PCI memory unmapped at 0x2200020000 EAL: PCI device 0000:b1:01.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_iavf EAL: Using IOMMU type 1 (Type 1) EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered EAL: Installed memory event callback for VFIO EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x2200024000 EAL: PCI memory mapped at 0x2200044000 EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:01.0 (socket 1) iavf_dev_init(): >> iavf_set_mac_type iavf_set_mac_type found mac: 2, returns: 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0 iavf_check_api_version(): Peer is supported PF host iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0 iavf_read_msg_from_pf(): AQ from pf carries opcode 17, retval 0 iavf_read_msg_from_pf(): Link status update:down iavf_read_msg_from_pf(): AQ from pf carries opcode 44, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 51, retval 0 iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26 EAL: PCI device 0000:b1:11.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_ice_dcf EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x2200048000 EAL: PCI memory mapped at 0x2200068000 EAL: Probe PCI driver: net_ice_dcf (8086:1889) device: 0000:b1:11.0 (socket 1) EAL: Releasing PCI mapped resource for 0000:b1:11.0 EAL: Calling pci_unmap_resource for 0000:b1:11.0 at 0x2200048000 EAL: PCI memory unmapped at 0x2200048000 EAL: Calling pci_unmap_resource for 0000:b1:11.0 at 0x2200068000 EAL: PCI memory unmapped at 0x2200068000 EAL: PCI device 0000:b1:11.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_iavf EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x220006c000 EAL: PCI memory mapped at 0x220008c000 EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:11.0 (socket 1) iavf_dev_init(): >> iavf_set_mac_type iavf_set_mac_type found mac: 2, returns: 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0 iavf_check_api_version(): Peer is supported PF host iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 17, retval 0 iavf_read_msg_from_pf(): Link status update:down iavf_read_msg_from_pf(): AQ from pf carries opcode 44, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 51, retval 0 iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26 EAL: lib.telemetry log level changed from disabled to debug TELEMETRY: Attempting socket bind to path '/var/run/dpdk/rte/dpdk_telemetry.v2' TELEMETRY: Socket creation and binding ok TELEMETRY: Telemetry initialized ok TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected testpmd: create a new mbuf pool <mb_pool_0>: n=155456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool <mb_pool_1>: n=155456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 1) iavf_handle_virtchnl_msg(): adminq response is received, opcode = 55 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23 iavf_dev_tx_queue_setup(): >> check_tx_vec_allow(): Vector tx can be enabled on this txq. iavf_dev_rx_queue_setup(): >> iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0. check_rx_vec_allow(): Vector Rx can be enabled on this rxq. iavf_dev_start(): >> iavf_set_rx_function(): Using avx2 Vector Rx (port 0). iavf_set_tx_function(): Using avx2 Vector Tx (port 0). iavf_configure_queues(): request RXDID[22] in Queue[0] iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7 iavf_add_del_all_mac_addr(): add/rm mac:00:DE:AD:01:01:01 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10 iavf_dev_tx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 0: link state change event iavf_dev_rx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 0: link state change event iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 Port 0: 00:DE:AD:01:01:01 Configuring Port 1 (socket 1) iavf_handle_virtchnl_msg(): adminq response is received, opcode = 55 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23 iavf_dev_tx_queue_setup(): >> check_tx_vec_allow(): Vector tx can be enabled on this txq. iavf_dev_rx_queue_setup(): >> iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0. check_rx_vec_allow(): Vector Rx can be enabled on this rxq. iavf_dev_start(): >> iavf_set_rx_function(): Using avx2 Vector Rx (port 1). iavf_set_tx_function(): Using avx2 Vector Tx (port 1). iavf_configure_queues(): request RXDID[22] in Queue[0] iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7 iavf_add_del_all_mac_addr(): add/rm mac:00:DE:AD:02:02:02 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10 iavf_dev_tx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 1: link state change event iavf_dev_rx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 1: link state change event iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 Port 1: 00:DE:AD:02:02:02 Checking link statuses... Done iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 testpmd>
Run same test on same environment, start testpmd with rhel8.6 + dpdk-21.11-3.el8 can work as well. [root@653cc75f4876 /]# dpdk-testpmd --log-level='*:debug' -l 3,25,23 -n 4 -m 1024 -- -i --forward-mode=mac --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --mbcache=512 --auto-start EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: lib.rcu log level changed from error to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: bus.vdev log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.22.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.22.0 EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.22.0 EAL: Registered [auxiliary] bus. EAL: bus.auxiliary log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.22.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.22.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.22.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.22.0 EAL: pmd.common.iavf log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.22.0 EAL: pmd.common.mlx5 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.22.0 EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.22.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.22.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.22.0 EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.22.0 EAL: pmd.net.ice.init log level changed from disabled to debug EAL: pmd.net.ice.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.22.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.22.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.22.0 EAL: pmd.net.mlx5 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.22.0 rte_vmbus_register(): Registered driver net_netvsc EAL: pmd.net.netvsc.init log level changed from disabled to debug EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.22.0 EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.22.0 EAL: pmd.net.qede.init log level changed from disabled to debug EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.22.0 EAL: Registered [vdpa] device class. EAL: lib.vhost.config log level changed from disabled to debug EAL: lib.vhost.data log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.22.0 EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: VFIO PCI modules not loaded EAL: PCI driver net_iavf for device 0000:b1:01.0 wants IOVA as 'DC' EAL: PCI driver net_ice_dcf for device 0000:b1:01.0 wants IOVA as 'DC' EAL: PCI driver net_iavf for device 0000:b1:11.0 wants IOVA as 'DC' EAL: PCI driver net_ice_dcf for device 0000:b1:11.0 wants IOVA as 'DC' EAL: Bus pci wants IOVA as 'DC' EAL: Bus vdev wants IOVA as 'DC' EAL: Bus auxiliary wants IOVA as 'DC' EAL: Buses did not request a specific IOVA mode. EAL: IOMMU is available, selecting IOVA as VA mode. EAL: Selected IOVA mode 'VA' EAL: No available 2048 kB hugepages reported EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is supported EAL: IOMMU type 7 (sPAPR) is not supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: VFIO support initialized EAL: Ask a virtual area of 0x2e000 bytes EAL: Virtual area found at 0x100007000 (size = 0x2e000) EAL: Setting up physically contiguous memory... EAL: Setting maximum number of open files to 1048576 EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824 EAL: Detected memory type: socket_id:1 hugepage_sz:1073741824 EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x100035000 (size = 0x1000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x140000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x140000000, size 800000000 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x940000000 (size = 0x1000) EAL: Memseg list allocated at socket 0, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x980000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x980000000, size 800000000 EAL: Creating 2 segment lists: n_segs:32 socket_id:1 hugepage_sz:1073741824 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x1180000000 (size = 0x1000) EAL: Memseg list allocated at socket 1, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x11c0000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x11c0000000, size 800000000 EAL: Ask a virtual area of 0x1000 bytes EAL: Virtual area found at 0x19c0000000 (size = 0x1000) EAL: Memseg list allocated at socket 1, page size 0x100000kB EAL: Ask a virtual area of 0x800000000 bytes EAL: Virtual area found at 0x1a00000000 (size = 0x800000000) EAL: VA reserved for memseg list at 0x1a00000000, size 800000000 EAL: Allocating 1 pages of size 1024M on socket 1 EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 1 EAL: Restoring previous memory policy: 0 EAL: Added 1024M to heap on socket 1 EAL: TSC frequency is ~2900000 KHz EAL: Main lcore 3 is ready (tid=7eff3ecb6900;cpuset=[3]) EAL: lcore 23 is ready (tid=7eff34e57700;cpuset=[23]) EAL: lcore 25 is ready (tid=7eff34656700;cpuset=[25]) EAL: PCI device 0000:b1:00.0 on NUMA socket 1 EAL: probe driver: 8086:1592 net_ice EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:b1:00.1 on NUMA socket 1 EAL: probe driver: 8086:1592 net_ice EAL: Not managed by a supported kernel driver, skipped EAL: PCI device 0000:b1:01.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_iavf EAL: Using IOMMU type 1 (Type 1) EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered EAL: Installed memory event callback for VFIO EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x2200000000 EAL: PCI memory mapped at 0x2200020000 EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:01.0 (socket 1) iavf_dev_init(): >> iavf_set_mac_type iavf_set_mac_type found mac: 2, returns: 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0 iavf_check_api_version(): Peer is supported PF host iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0 iavf_read_msg_from_pf(): AQ from pf carries opcode 17, retval 0 iavf_read_msg_from_pf(): Link status update:down iavf_read_msg_from_pf(): AQ from pf carries opcode 44, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 51, retval 0 iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26 EAL: PCI device 0000:b1:11.0 on NUMA socket 1 EAL: probe driver: 8086:1889 net_iavf EAL: VFIO reports MSI-X BAR as mappable EAL: PCI memory mapped at 0x2200024000 EAL: PCI memory mapped at 0x2200044000 EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:11.0 (socket 1) iavf_dev_init(): >> iavf_set_mac_type iavf_set_mac_type found mac: 2, returns: 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0 iavf_check_api_version(): Peer is supported PF host iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 17, retval 0 iavf_read_msg_from_pf(): Link status update:down iavf_read_msg_from_pf(): AQ from pf carries opcode 44, retval 0 iavf_read_msg_from_pf(): Can't read msg from AQ iavf_read_msg_from_pf(): AQ from pf carries opcode 51, retval 0 iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 26 EAL: lib.telemetry log level changed from disabled to debug TELEMETRY: Attempting socket bind to path '/var/run/dpdk/rte/dpdk_telemetry.v2' TELEMETRY: Socket creation and binding ok TELEMETRY: Telemetry initialized ok TELEMETRY: No legacy callbacks, legacy socket not created Interactive-mode selected Set mac packet forwarding mode Auto-start selected testpmd: create a new mbuf pool <mb_pool_1>: n=180224, size=9728, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc EAL: Trying to obtain current memory policy. EAL: Setting policy MPOL_PREFERRED for socket 1 EAL: Restoring previous memory policy: 0 EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)' EAL: request: mp_malloc_sync EAL: Heap on socket 1 was expanded by 2048MB Configuring Port 0 (socket 1) iavf_handle_virtchnl_msg(): adminq response is received, opcode = 55 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23 iavf_dev_tx_queue_setup(): >> check_tx_vec_allow(): Vector tx can be enabled on this txq. iavf_dev_rx_queue_setup(): >> iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0. check_rx_vec_allow(): Vector Rx can be enabled on this rxq. iavf_dev_start(): >> iavf_set_rx_function(): Using avx2 Vector Rx (port 0). iavf_set_tx_function(): Using avx2 Vector Tx (port 0). iavf_configure_queues(): request RXDID[22] in Queue[0] iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7 iavf_add_del_all_mac_addr(): add/rm mac:00:DE:AD:01:01:01 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10 iavf_dev_tx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 0: link state change event iavf_dev_rx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 0: link state change event iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 Port 0: 00:DE:AD:01:01:01 Configuring Port 1 (socket 1) iavf_handle_virtchnl_msg(): adminq response is received, opcode = 55 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 57 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 23 iavf_dev_tx_queue_setup(): >> check_tx_vec_allow(): Vector tx can be enabled on this txq. iavf_dev_rx_queue_setup(): >> iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0. check_rx_vec_allow(): Vector Rx can be enabled on this rxq. iavf_dev_start(): >> iavf_set_rx_function(): Using avx2 Vector Rx (port 1). iavf_set_tx_function(): Using avx2 Vector Tx (port 1). iavf_configure_queues(): request RXDID[22] in Queue[0] iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7 iavf_add_del_all_mac_addr(): add/rm mac:00:DE:AD:02:02:02 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10 iavf_dev_tx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 1: link state change event iavf_dev_rx_queue_start(): >> iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8 iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event Port 1: link state change event iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 Port 1: 00:DE:AD:02:02:02 Checking link statuses... Done iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14 Start automatic packet forwarding mac packet forwarding - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native Logical Core 23 (socket 1) forwards packets on 1 streams: RX P=0/Q=0 (socket 1) -> TX P=1/Q=0 (socket 1) peer=00:00:00:00:00:02 Logical Core 25 (socket 1) forwards packets on 1 streams: RX P=1/Q=0 (socket 1) -> TX P=0/Q=0 (socket 1) peer=00:00:00:00:00:01 mac packet forwarding packets/burst=32 nb forwarding cores=2 - nb forwarding ports=2 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x10000 RX queue: 0 RX desc=4096 - RX free threshold=32 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=4096 - TX free threshold=32 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x10000 - TX RS bit threshold=32 port 1: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x10000 RX queue: 0 RX desc=4096 - RX free threshold=32 RX threshold registers: pthresh=0 hthresh=0 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=4096 - TX free threshold=32 TX threshold registers: pthresh=0 hthresh=0 wthresh=0 TX offloads=0x10000 - TX RS bit threshold=32 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15 iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15 testpmd>
- I am not able to reproduce this issue. I am missing the way to generate the container image and I am not sure how the 21.11 container is started. - The results between v21.11 and v22.11 can't be compared. The commandlines are different. The main difference I noticed is around the core list: v21.11 is using an explicit corelist on cores 3,25,23, while v22.11 is using none. Looking at the v22.11 failing test: > EAL: Failed to create thread for interrupt handling > EAL: FATAL: Cannot init interrupt-handling thread > EAL: Cannot init interrupt-handling thread > EAL: Error - exiting with code: 1 > Cause: Cannot init EAL: Invalid argument This is likely related to cpu affinity for this thread. Please install the glibc-debuginfo and gdb packages inside the container and run the following commands: # dnf install -y glibc-debuginfo gdb # taskset -pc $$ # cat > commands.gdb <<EOF set breakpoint pending on b pthread_setaffinity_np commands 1 print *cpuset continue end run --log-level="*:debug" -a 0000:b1:01.0 -a 0000:b1:11.0 -n 4 -m 1024 -- -i EOF # gdb $(which dpdk-testpmd) --command commands.gdb
Here is start podman commands [root@dell-per750-13 ~]# wget -nv -N -c -t 3 http://netqe-bj.usersys.redhat.com/share/mhou/image/container/rhel9_x86_64 2023-03-27 08:13:40 URL:http://netqe-bj.usersys.redhat.com/share/mhou/image/container/rhel9_x86_64 [1279621632/1279621632] -> "rhel9_x86_64" [1] [root@dell-per750-13 ~]# podman load -i rhel9_x86_64 Getting image source signatures Copying blob 32198741d699 done Copying blob 9776ad2a4dfd done Copying blob eaa63378b3b0 done Copying blob 2746550f79cc done Copying blob 88db4a29029b done Copying blob 589ed0bccb42 done Copying config 8d236f403e done Writing manifest to image destination Storing signatures Loaded image: localhost/rhel9.0_x86_64:latest [root@dell-per750-13 ~]# podman images -a REPOSITORY TAG IMAGE ID CREATED SIZE localhost/rhel9.0_x86_64 latest 8d236f403e2c 3 weeks ago 1.28 GB [root@dell-per750-13 ~]# podman run --name=test1 -dt --privileged --cpuset-mems=1 --cpuset-cpus=3,25,23 -v /dev/vfio/vfio:/dev/vfio/vfio -v /dev/hugepages:/dev/hugepages 8d236f403e2c /bin/bash update yum repo in container for i in $(ls /etc/yum.repos.d/*.repo); do podman container cp ${i} test1:/etc/yum.repos.d/ done install dpdk/dpdk-tools/glibc-debuginfo gdb # taskset -pc $$ pid 107's current affinity list: 0,1 cat > commands.gdb <<EOF set breakpoint pending on b pthread_setaffinity_np commands 1 print *cpuset continue end run --log-level="*:debug" -a 0000:b1:01.0 -a 0000:b1:11.0 -n 4 -m 1024 -- -i EOF [root@73c165597ca6 /]# gdb $(which dpdk-testpmd) --command commands.gdb GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/dpdk-testpmd... Reading symbols from .gnu_debugdata for /usr/bin/dpdk-testpmd... (No debugging symbols found in .gnu_debugdata for /usr/bin/dpdk-testpmd) Function "pthread_setaffinity_np" not defined. Breakpoint 1 (pthread_setaffinity_np) pending. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: bus.auxiliary log level changed from notice to debug EAL: lib.rcu log level changed from error to debug EAL: pmd.common.mlx5 log level changed from notice to debug EAL: pmd.common.iavf log level changed from notice to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: pmd.net.mlx5 log level changed from notice to debug EAL: pmd.net.ice.driver log level changed from notice to debug EAL: pmd.net.ice.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.23.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.23.0 EAL: Registered [vdev] bus. EAL: bus.vdev log level changed from disabled to debug EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.23.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.23.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.23.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.23.0 EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.23.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.23.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.23.0 EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.23.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.23.0 EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: pmd.net.netvsc.init log level changed from disabled to debug rte_vmbus_register(): Registered driver net_netvsc EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.23.0 EAL: pmd.net.nfp.cpp log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.23.0 EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: pmd.net.qede.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.23.0 EAL: lib.dmadev log level changed from disabled to debug EAL: Registered [vdpa] device class. EAL: lib.vhost.data log level changed from disabled to debug EAL: lib.vhost.config log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.23.0 EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) [New Thread 0x7ffff6b00640 (LWP 231)] [Switching to Thread 0x7ffff6b00640 (LWP 231)] Thread 2 "eal-intr-thread" hit Breakpoint 1, 0x00007ffff7148310 in pthread_setaffinity_np.4 () from /lib64/libc.so.6 commands.gdb:7: Error in sourced command file: No symbol "cpuset" in current context. Missing separate debuginfos, use: dnf debuginfo-install dpdk-22.11-1.el9.x86_64 (gdb)
After install debuginfo packages, re-run same command and the output as below: [root@73c165597ca6 /]# gdb $(which dpdk-testpmd) --command commands.gdb GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/dpdk-testpmd... Reading symbols from /usr/lib/debug/usr/bin/dpdk-testpmd-22.11-1.el9.x86_64.debug... Function "pthread_setaffinity_np" not defined. Breakpoint 1 (pthread_setaffinity_np) pending. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: bus.auxiliary log level changed from notice to debug EAL: lib.rcu log level changed from error to debug EAL: pmd.common.mlx5 log level changed from notice to debug EAL: pmd.common.iavf log level changed from notice to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: pmd.net.mlx5 log level changed from notice to debug EAL: pmd.net.ice.driver log level changed from notice to debug EAL: pmd.net.ice.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.23.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.23.0 EAL: Registered [vdev] bus. EAL: bus.vdev log level changed from disabled to debug EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.23.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.23.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.23.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.23.0 EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.23.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.23.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.23.0 EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.23.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.23.0 EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: pmd.net.netvsc.init log level changed from disabled to debug rte_vmbus_register(): Registered driver net_netvsc EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.23.0 EAL: pmd.net.nfp.cpp log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.23.0 EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: pmd.net.qede.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.23.0 EAL: lib.dmadev log level changed from disabled to debug EAL: Registered [vdpa] device class. EAL: lib.vhost.data log level changed from disabled to debug EAL: lib.vhost.config log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.23.0 EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) [New Thread 0x7ffff6af3640 (LWP 497)] [Switching to Thread 0x7ffff6af3640 (LWP 497)] Thread 2 "eal-intr-thread" hit Breakpoint 1, __pthread_setaffinity_new (th=140737332065856, cpusetsize=cpusetsize@entry=128, cpuset=cpuset@entry=0x7ffff7d2a428 <internal_config.lto_priv+12712>) at pthread_setaffinity.c:29 29 { $1 = {__bits = {1, 0 <repeats 15 times>}} EAL: Failed to create thread for interrupt handling [Thread 0x7ffff6af3640 (LWP 497) exited] EAL: FATAL: Cannot init interrupt-handling thread EAL: Cannot init interrupt-handling thread EAL: Error - exiting with code: 1 Cause: Cannot init EAL: Invalid argument [Inferior 1 (process 493) exited with code 01] (gdb)
specify cpu list when start testpmd on container [root@73c165597ca6 /]# cat commands.gdb set breakpoint pending on b pthread_setaffinity_np commands 1 print *cpuset continue end run --log-level="*:debug" -l 3,25,23 -a 0000:b1:01.0 -a 0000:b1:11.0 -n 4 -m 1024 -- -i [root@73c165597ca6 /]# gdb $(which dpdk-testpmd) --command commands.gdb GNU gdb (GDB) Red Hat Enterprise Linux 10.2-10.el9 Copyright (C) 2021 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/dpdk-testpmd... Reading symbols from /usr/lib/debug/usr/bin/dpdk-testpmd-22.11-1.el9.x86_64.debug... Function "pthread_setaffinity_np" not defined. Breakpoint 1 (pthread_setaffinity_np) pending. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". EAL: lib.eal log level changed from info to debug EAL: lib.malloc log level changed from info to debug EAL: lib.ring log level changed from info to debug EAL: lib.mempool log level changed from info to debug EAL: lib.timer log level changed from info to debug EAL: pmd log level changed from info to debug EAL: lib.hash log level changed from info to debug EAL: lib.lpm log level changed from info to debug EAL: lib.kni log level changed from info to debug EAL: lib.acl log level changed from info to debug EAL: lib.power log level changed from info to debug EAL: lib.meter log level changed from info to debug EAL: lib.sched log level changed from info to debug EAL: lib.port log level changed from info to debug EAL: lib.table log level changed from info to debug EAL: lib.pipeline log level changed from info to debug EAL: lib.mbuf log level changed from info to debug EAL: lib.cryptodev log level changed from info to debug EAL: lib.efd log level changed from info to debug EAL: lib.eventdev log level changed from info to debug EAL: lib.gso log level changed from info to debug EAL: user1 log level changed from info to debug EAL: user2 log level changed from info to debug EAL: user3 log level changed from info to debug EAL: user4 log level changed from info to debug EAL: user5 log level changed from info to debug EAL: user6 log level changed from info to debug EAL: user7 log level changed from info to debug EAL: user8 log level changed from info to debug EAL: bus.auxiliary log level changed from notice to debug EAL: lib.rcu log level changed from error to debug EAL: pmd.common.mlx5 log level changed from notice to debug EAL: pmd.common.iavf log level changed from notice to debug EAL: lib.net log level changed from info to debug EAL: lib.ethdev log level changed from info to debug EAL: pmd.net.mlx5 log level changed from notice to debug EAL: pmd.net.ice.driver log level changed from notice to debug EAL: pmd.net.ice.init log level changed from notice to debug EAL: pmd.net.ixgbe.driver log level changed from notice to debug EAL: pmd.net.ixgbe.init log level changed from notice to debug EAL: pmd.net.i40e.driver log level changed from notice to debug EAL: pmd.net.i40e.init log level changed from notice to debug EAL: pmd.net.bnxt.driver log level changed from notice to debug EAL: lib.bpf log level changed from info to debug EAL: lib.pdump log level changed from notice to debug EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 1 EAL: Detected lcore 2 as core 8 on socket 0 EAL: Detected lcore 3 as core 8 on socket 1 EAL: Detected lcore 4 as core 1 on socket 0 EAL: Detected lcore 5 as core 1 on socket 1 EAL: Detected lcore 6 as core 9 on socket 0 EAL: Detected lcore 7 as core 9 on socket 1 EAL: Detected lcore 8 as core 2 on socket 0 EAL: Detected lcore 9 as core 2 on socket 1 EAL: Detected lcore 10 as core 10 on socket 0 EAL: Detected lcore 11 as core 10 on socket 1 EAL: Detected lcore 12 as core 3 on socket 0 EAL: Detected lcore 13 as core 3 on socket 1 EAL: Detected lcore 14 as core 11 on socket 0 EAL: Detected lcore 15 as core 11 on socket 1 EAL: Detected lcore 16 as core 4 on socket 0 EAL: Detected lcore 17 as core 4 on socket 1 EAL: Detected lcore 18 as core 12 on socket 0 EAL: Detected lcore 19 as core 12 on socket 1 EAL: Detected lcore 20 as core 5 on socket 0 EAL: Detected lcore 21 as core 5 on socket 1 EAL: Detected lcore 22 as core 13 on socket 0 EAL: Detected lcore 23 as core 13 on socket 1 EAL: Detected lcore 24 as core 6 on socket 0 EAL: Detected lcore 25 as core 6 on socket 1 EAL: Detected lcore 26 as core 14 on socket 0 EAL: Detected lcore 27 as core 14 on socket 1 EAL: Detected lcore 28 as core 7 on socket 0 EAL: Detected lcore 29 as core 7 on socket 1 EAL: Detected lcore 30 as core 15 on socket 0 EAL: Detected lcore 31 as core 15 on socket 1 EAL: Maximum logical cores by configuration: 128 EAL: Detected CPU lcores: 32 EAL: Detected NUMA nodes: 2 EAL: Checking presence of .so 'librte_eal.so.23.0' EAL: Detected shared linkage of DPDK EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.23.0 EAL: Registered [vdev] bus. EAL: bus.vdev log level changed from disabled to debug EAL: pmd.net.ring log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_auxiliary.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.23.0 EAL: pmd.net.tap log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.23.0 EAL: pmd.net.vdev_netvsc log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.23.0 EAL: Registered [vmbus] bus. EAL: bus.vmbus log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.23.0 EAL: pmd.net.e1000.driver log level changed from disabled to debug EAL: pmd.net.e1000.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.23.0 EAL: pmd.net.enic log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.23.0 EAL: pmd.net.failsafe log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.23.0 EAL: pmd.net.iavf.driver log level changed from disabled to debug EAL: pmd.net.iavf.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.23.0 EAL: pmd.net.mlx4 log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.23.0 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.23.0 EAL: pmd.net.netvsc.driver log level changed from disabled to debug EAL: pmd.net.netvsc.init log level changed from disabled to debug rte_vmbus_register(): Registered driver net_netvsc EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.23.0 EAL: pmd.net.nfp.cpp log level changed from disabled to debug EAL: pmd.net.nfp.driver log level changed from disabled to debug EAL: pmd.net.nfp.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.23.0 EAL: pmd.net.qede.driver log level changed from disabled to debug EAL: pmd.net.qede.init log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.23.0 EAL: lib.dmadev log level changed from disabled to debug EAL: Registered [vdpa] device class. EAL: lib.vhost.data log level changed from disabled to debug EAL: lib.vhost.config log level changed from disabled to debug EAL: pmd.net.vhost log level changed from disabled to debug EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.23.0 EAL: pmd.net.virtio.driver log level changed from disabled to debug EAL: pmd.net.virtio.init log level changed from disabled to debug EAL: Ask a virtual area of 0x7000 bytes EAL: Virtual area found at 0x100000000 (size = 0x7000) [New Thread 0x7ffff6af3640 (LWP 541)] [Switching to Thread 0x7ffff6af3640 (LWP 541)] Thread 2 "eal-intr-thread" hit Breakpoint 1, __pthread_setaffinity_new (th=140737332065856, cpusetsize=cpusetsize@entry=128, cpuset=cpuset@entry=0x7ffff7d2a428 <internal_config.lto_priv+12712>) at pthread_setaffinity.c:29 29 { $1 = {__bits = {3, 0 <repeats 15 times>}} EAL: Failed to create thread for interrupt handling EAL: FATAL: Cannot init interrupt-handling thread EAL: Cannot init interrupt-handling thread [Thread 0x7ffff6af3640 (LWP 541) exited] EAL: Error - exiting with code: 1 Cause: Cannot init EAL: Invalid argument [Inferior 1 (process 537) exited with code 01] (gdb)
(In reply to mhou from comment #3) > # taskset -pc $$ > pid 107's current affinity list: 0,1 We can see that, from inside the container, only cores 0 and 1 are reported available. I would expect 3, 23 and 25 (since the container was created with "--cpuset-cpus=3,25,23"). This is where something is wrong. (In reply to mhou from comment #4) > Thread 2 "eal-intr-thread" hit Breakpoint 1, __pthread_setaffinity_new > (th=140737332065856, cpusetsize=cpusetsize@entry=128, > cpuset=cpuset@entry=0x7ffff7d2a428 <internal_config.lto_priv+12712>) at > pthread_setaffinity.c:29 > 29 { > $1 = {__bits = {1, 0 <repeats 15 times>}} > EAL: Failed to create thread for interrupt handling This gdb trace confirms it. DPDK tried to pin the interrupt thread to core 0 (which is the first available core from the current cpu affinity). In this same container, I suppose running the following command fails, can you check? # taskset -c 0 echo plop With your image, I can start a container on a RHEL 8 system (podman-4.2.0-8.module+el8.7.0+17824+66a0202b.x86_64). [root@wsfd-netdev82 ~]# podman run --name=test1 -dt --privileged --cpuset-mems=0 --cpuset-cpus=3,25,23 -v /dev/vfio/vfio:/dev/vfio/vfio -v /dev/hugepages:/dev/hugepages 8d236f403e2c /bin/bash 91faf38fe33beb1efab81f86800b3d7d0b3948d9c0be1997f84c101beeb16337 [root@wsfd-netdev82 ~]# podman exec -it test1 /bin/bash [root@91faf38fe33b /]# taskset -pc $$ pid 17's current affinity list: 3,23,25 And dpdk starts fine. I am not sure what would result in this 0,1 cpu affinity in the container. The issue is likely in RHEL9 podman / the container engine, or in their running environment. One first idea, did you apply some cpuaffinity to the podman / containers daemons? Another idea, could you try to downgrade podman and see if this is a podman regression?
Hello David You are right. I will debug and open a new issue to container.
*** This bug has been marked as a duplicate of bug 2182238 ***
I don't think we needed a new bz, you could have moved to a different component/product. In any case, as a workaround, you can start your application through an explicit taskset in the container. Like: # taskset -c 3,25,23 dpdk-testpmd etc...