Bug 2148797

Summary:	mlx5_core driver: testpmd start failed inside guest when running sriov dpdk pvp case
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	liting <tli>
Component:	DPDK	Assignee:	David Marchand <dmarchan>
DPDK sub component:	sriov	QA Contact:	liting <tli>
Status:	NEW ---	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	ctrautma, dmarchan, hewang, jasowang, jhsiao
Version:	FDP 22.K	Flags:	tli: needinfo? (jasowang)
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description liting 2022-11-28 01:56:28 UTC

Description of problem:


Version-Release number of selected component (if applicable):
host and guest os are use rhel9.2
kernel: 5.14.0-197.el9.x86_64
dpdk-21.11.2-1.el9_1.x86_64.rpm

How reproducible:

Steps to Reproduce:
1. create 1 vf for per pf
2. bind the vf port to vfio-pci
3. create guest with the two vfs, the guest xml as follows
<domain type='kvm'>
  <name>g1</name>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <locked/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='26'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <emulatorpin cpuset='0,24'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/g1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
  </source>
  <mac address='00:de:ad:01:01:01'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
  </source>
  <mac address='00:de:ad:02:02:02'/>
</hostdev>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>

4. start testpmd inside guest:
dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start

Actual results:
testpmd start failed inside guest:
EAL: Detected CPU lcores: 3
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 2048 kB hugepages reported
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:02:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:03:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:03:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:04:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:04:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:04:00.0 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0
EAL: Error - exiting with code: 1
  Cause: rxq 1 invalid - must be >= 0 && <= 0
Interactive-mode selected
Set mac packet forwarding mode

Expected results:
Testpmd start successfully inside guest.

Additional info:
rhel9.2: testpmd start failed inside guest
https://beaker.engineering.redhat.com/jobs/7280738
rhel9.1: work well
https://beaker.engineering.redhat.com/jobs/7282526
rhel9.0: work well
https://beaker.engineering.redhat.com/jobs/7278158
rhel8.8: work well
https://beaker.engineering.redhat.com/jobs/7281676

Comment 1 liting 2022-11-29 03:04:15 UTC

Debug it and found this root cause. when guest using rhel9.2 kernel(5.14.0-197.el9.x86_64), if run following steps and reboot guest, there is no the two vfs port inside guest. so the testpmd start failed inside guest. When guest using rhel9.0 or rhel9.1, host use rhel9.2, it still has this issue. 
For rhel9.2 host, run following command.
[root@dell-per730-56 perf]# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/0000:04:00.1/sriov_numvfs
[root@dell-per730-56 perf]# echo 1 > /sys/devices/pci0000:00/0000:00:02.0/0000:04:00.0/sriov_numvfs
[root@dell-per730-56 perf]# driverctl -v list-overrides
driverctl: No overridable devices found. Kernel too old?
[root@dell-per730-56 perf]# cat g1.xml 
<domain type='kvm'>
  <name>g1</name>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <locked/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='26'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <emulatorpin cpuset='0,24'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/g1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
  </source>
  <mac address='00:de:ad:01:01:01'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x0'/>
  </source>
  <mac address='00:de:ad:02:02:02'/>
</hostdev>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>
[root@dell-per730-56 perf]# virsh define g1.xml 
Domain 'g1' defined from g1.xml
[root@dell-per730-56 perf]# driverctl -v list-overrides
driverctl: No overridable devices found. Kernel too old?
[root@dell-per730-56 perf]# virsh start g1
Domain 'g1' started

[root@dell-per730-56 perf]# driverctl -v list-overrides
0000:04:00.2 vfio-pci (MT27800 Family [ConnectX-5 Virtual Function])
0000:04:01.0 vfio-pci (MT27800 Family [ConnectX-5 Virtual Function])

[root@dell-per730-56 perf]# virsh console g1
Connected to domain 'g1'
Escape character is ^] (Ctrl + ])

localhost login: root
Password: 
Last login: Mon Nov 28 20:46:53 on ttyS0
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3525sec preferred_lft 3525sec
    inet6 fe80::f9af:8f13:60f3:5749/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e2:5e:b5:d5:7a:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a465:f48d:9287:f487/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 32:23:ad:a6:2c:c2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::52a9:78c6:aac6:aaf5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

[root@localhost ~]# reboot
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3583sec preferred_lft 3583sec
    inet6 fe80::3f78:cc40:6622:60af/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# uname -r
5.14.0-197.el9.x86_64
[root@localhost ~]# dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start
EAL: Detected CPU lcores: 3
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 2048 kB hugepages reported
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:02:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:03:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:03:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:04:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:04:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:04:00.0 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0
EAL: Error - exiting with code: 1
  Cause: rxq 1 invalid - must be >= 0 && <= 0
Interactive-mode selected
Set mac packet forwarding mode

Comment 2 liting 2022-11-29 03:06:05 UTC

When guest use the latest version 5.14.0-200.el9.x86_64, it has no this issue.

Comment 3 Hekai Wang 2022-12-01 06:55:41 UTC

same issue encountered in rhel8.8
Here are some jobs 
ovs dpdk bonding
i40e https://beaker.engineering.redhat.com/jobs/7287561
ice https://beaker.engineering.redhat.com/jobs/7287562
mlx5_core https://beaker.engineering.redhat.com/jobs/7287563
mlx5-cx6 https://beaker.engineering.redhat.com/jobs/7287564

Comment 4 Hekai Wang 2022-12-01 09:18:23 UTC

With kernel kernel-5.14.0-201.el9, still has this issue . 
Here is the job 
https://beaker.engineering.redhat.com/jobs/7290107

Comment 5 liting 2022-12-21 01:43:48 UTC

kernel-5.14.0-217.el9 still has this issue.
https://beaker.engineering.redhat.com/jobs/7363528

Comment 6 liting 2023-01-29 06:19:41 UTC

RHEL 9.2 CTC2 still has this issue
https://beaker.engineering.redhat.com/jobs/7476517

Comment 7 David Marchand 2023-01-30 12:43:16 UTC

(In reply to liting from comment #0)
> 4. start testpmd inside guest:
> dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac
> --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728
> --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01
> --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start
> 
> Actual results:
> testpmd start failed inside guest:
> EAL: Detected CPU lcores: 3
> EAL: Detected NUMA nodes: 1
> EAL: Detected shared linkage of DPDK
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'VA'
> EAL: No available 2048 kB hugepages reported
> EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0)
> eth_virtio_pci_init(): Failed to init PCI device
> EAL: Requested device 0000:02:00.0 cannot be used
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:00.0 (socket 0)
> mlx5_common: Verbs device not found: 0000:03:00.0
> mlx5_common: Failed to initialize device context.
> EAL: Requested device 0000:03:00.0 cannot be used
> EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:04:00.0 (socket 0)
> mlx5_common: Verbs device not found: 0000:04:00.0
> mlx5_common: Failed to initialize device context.
> EAL: Requested device 0000:04:00.0 cannot be used
> TELEMETRY: No legacy callbacks, legacy socket not created
> testpmd: No probed ethernet devices
> Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0
> EAL: Error - exiting with code: 1
>   Cause: rxq 1 invalid - must be >= 0 && <= 0
> Interactive-mode selected
> Set mac packet forwarding mode

I can't reproduce this issue.

I installed a fresh rhel9.2 on the host.
[root@wsfd-netdev18 ~]# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-244.el9.x86_64 root=/dev/mapper/rhel_wsfd--netdev18-root ro default_hugepagesz=1G hugepagesz=1G hugepages=32 iommu=pt intel_iommu=on crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/rhel_wsfd--netdev18-swap rd.lvm.lv=rhel_wsfd-netdev18/root rd.lvm.lv=rhel_wsfd-netdev18/swap console=ttyS1,115200
[root@wsfd-netdev18 ~]# rpm -q qemu-kvm
qemu-kvm-7.2.0-5.el9.x86_64
[root@wsfd-netdev18 ~]# driverctl -v list-overrides
0000:04:00.2 vfio-pci (MT28800 Family [ConnectX-5 Ex Virtual Function])
0000:04:01.2 vfio-pci (MT28800 Family [ConnectX-5 Ex Virtual Function])
[root@wsfd-netdev18 ~]# cat g1.xml 
<domain type='kvm'>
  <name>g1</name>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <locked/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <emulatorpin cpuset='0,6'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pmu state='off'/>
    <vmport state='off'/>
    <ioapic driver='qemu'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/root/g1.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='none'>
      <alias name='usb'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x8'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x9'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xa'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
    </interface>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
  </source>
  <mac address='00:de:ad:01:01:01'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address type='pci' domain='0x0000' bus='0x04' slot='0x01' function='0x2'/>
  </source>
  <mac address='00:de:ad:02:02:02'/>
</hostdev>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <iommu model='intel'>
      <driver intremap='on' caching_mode='on' iotlb='on'/>
    </iommu>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'/>
</domain>


The guest is a RHEL 9.2 too, using the same beaker repo than the host.
[root@g1 ~]# lspci | grep Ethernet
02:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
03:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
04:00.0 Ethernet controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex Virtual Function]
(failed reverse-i-search)`etstpmd': lspci | grep Ethern^C
[root@g1 ~]# dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start
EAL: Detected CPU lcores: 3
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 1048576 kB hugepages reported
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:02:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:101a) device: 0000:03:00.0 (socket 0)
EAL: Probe PCI driver: mlx5_pci (15b3:101a) device: 0000:04:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set mac packet forwarding mode
Auto-start selected
testpmd: create a new mbuf pool <mb_pool_0>: n=180224, size=9728, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 0)
Port 0: 7E:4B:A6:2E:73:F8
Configuring Port 1 (socket 0)
Port 1: 0E:C1:71:8B:23:1D
Checking link statuses...
Done
Start automatic packet forwarding
mac packet forwarding - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 0) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=00:00:00:00:00:02
Logical Core 2 (socket 0) forwards packets on 1 streams:
  RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=00:00:00:00:00:01

  mac packet forwarding packets/burst=32
  nb forwarding cores=2 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=4096 - RX free threshold=64
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=4096 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=0
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=4096 - RX free threshold=64
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=4096 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=0
testpmd> Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
Done

Stopping port 1...
Stopping ports...
Done

Shutting down port 0...
Closing ports...
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
Port 1 is closed
Done

Bye...




The beaker logs provide no usable information.
Please give access to this system with a reproducer.
Thanks.

Comment 8 liting 2023-01-31 04:03:54 UTC

I have the reproducer as follows,  when access to guest, and run "ip a" command, it can see enp3s0 and enp4s0 port, and then reboot the guest, and it cannot see enp3s0 and enp4s0 port anymore. Then run testpmd command, it can reproduce this issue. My test system is dell-per730-56.rhts.eng.pek2.redhat.com, you can access to it now. account/password is root/redhat.
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3525sec preferred_lft 3525sec
    inet6 fe80::f9af:8f13:60f3:5749/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e2:5e:b5:d5:7a:41 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a465:f48d:9287:f487/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 32:23:ad:a6:2c:c2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::52a9:78c6:aac6:aaf5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# reboot
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3583sec preferred_lft 3583sec
    inet6 fe80::3f78:cc40:6622:60af/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# dpdk-testpmd -l 0-2 -n 1 --socket-mem 1024 -- -i --forward-mode=mac --burst=32 --rxd=4096 --txd=4096 --max-pkt-len=9120 --mbuf-size=9728 --nb-cores=2 --rxq=1 --txq=1 --eth-peer=0,00:00:00:00:00:01 --eth-peer=1,00:00:00:00:00:02 --mbcache=512 --auto-start
EAL: Detected CPU lcores: 3
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 2048 kB hugepages reported
EAL: Probe PCI driver: net_virtio (1af4:1041) device: 0000:02:00.0 (socket 0)
eth_virtio_pci_init(): Failed to init PCI device
EAL: Requested device 0000:02:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:03:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:03:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:03:00.0 cannot be used
EAL: Probe PCI driver: mlx5_pci (15b3:1018) device: 0000:04:00.0 (socket 0)
mlx5_common: Verbs device not found: 0000:04:00.0
mlx5_common: Failed to initialize device context.
EAL: Requested device 0000:04:00.0 cannot be used
TELEMETRY: No legacy callbacks, legacy socket not created
testpmd: No probed ethernet devices
Fail: input rxq (1) can't be greater than max_rx_queues (0) of port 0
EAL: Error - exiting with code: 1
  Cause: rxq 1 invalid - must be >= 0 && <= 0
Interactive-mode selected
Set mac packet forwarding mode

Comment 9 David Marchand 2023-01-31 08:16:15 UTC

I can't reproduce.

Comment 10 David Marchand 2023-01-31 08:20:53 UTC

Sorry, I meant, that I can't reproduce on my system.
I'll have a look at your system.

Comment 11 David Marchand 2023-01-31 09:25:54 UTC

- I logged in the vm, and started testpmd, every rdma ioctl was *super* slow (I could see this via strace),
  but ended up initialising fine,

- after rebooting the vm, I can see in the vm dmesg that the infiniband kernel driver is *super* slow to initialise too:
[    6.615048] mlx5_core 0000:03:00.0: firmware version: 16.31.1014
[    6.848867] mlx5_core 0000:03:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[    7.159308] mlx5_core 0000:04:00.0: firmware version: 16.31.1014
[    7.352350] mlx5_core 0000:04:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[    9.681689] mlx5_core 0000:03:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[   90.681654] mlx5_core 0000:03:00.0: Supported tc offload range - chains: 1, prios: 1
[   98.682242] mlx5_core 0000:04:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  162.185663] mlx5_core 0000:04:00.0: Supported tc offload range - chains: 1, prios: 1
  After waiting 5 minutes that the kernel driver finished, I started testpmd which was super slow, but ended up initialising fine,

- I stopped the vm, the VF devices were returned to mlx5 kernel driver on the host.
  I started testpmd on those VF in the host, no issue, initialisation seems normal, no slowdown,

- I restarted the vm, kernel was ok, then started testpmd fine, no slowdown,

- I rebooted the vm, and we are back with the same phenomenon:
[    6.685098] mlx5_core 0000:03:00.0: firmware version: 16.31.1014
[    6.889649] mlx5_core 0000:03:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[    7.200112] mlx5_core 0000:04:00.0: firmware version: 16.31.1014
[    7.420885] mlx5_core 0000:04:00.0: Rate limit: 127 rates are supported, range: 0Mbps to 97656Mbps
[    9.685476] mlx5_core 0000:03:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  101.189469] mlx5_core 0000:03:00.0: Supported tc offload range - chains: 1, prios: 1
[  110.686708] mlx5_core 0000:04:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[  182.627968] mlx5_core 0000:04:00.0: Supported tc offload range - chains: 1, prios: 1


This slowdown in the guest is strange and likely the source of the reported issue.

qemu logs looks ok to me.
I recorded perf events on the CPU used by qemu, I can't conclude anything from it.
I see some kvm calls, and some idle task.
The host system itself seems rather idle.


I don't think I can help more.
I recommend contacting qemu/kvm people so they have a look at this system.



One last thing.

I noticed that there are VPD warnings on the host:
[root@dell-per730-56 ~]# dmesg | grep -i VPD
[    6.106574] pci 0000:03:00.0: [Firmware Bug]: disabling VPD access (can't determine size of non-standard VPD format)
[    6.118532] pci 0000:03:00.0: quirk_blacklist_vpd+0x0/0x22 took 11718 usecs
[  536.822487] pci 0000:82:00.0: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update
[  536.963396] pci 0000:82:00.1: VPD access failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update

Comment 12 jason wang 2023-02-01 04:23:58 UTC

Hi:

Could you please confirm whether the firmware is up-to-date? If not, please upgrate the firmware. If not, we can add NV engineers to have a look at this.

Thanks

Comment 13 liting 2023-02-01 07:52:09 UTC

(In reply to jason wang from comment #12)
> Hi:
> 
> Could you please confirm whether the firmware is up-to-date? If not, please
> upgrate the firmware. If not, we can add NV engineers to have a look at this.
> 
> Thanks

Hi, 

The firmware is 16.31.1014, the latest firmware maybe the 16.33.1048. I will try to update it to latest. I think it maybe not relate with the firmware. It work well when host is rhel9.0 or rhel9.1. It only has issue on rhel9.2 host.
[root@dell-per730-56 perf]# ethtool -i enp4s0f0np0
driver: mlx5_core
version: 5.14.0-197.el9.x86_64
firmware-version: 16.31.1014 (MT_0000000012)
expansion-rom-version: 
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

thanks,
Li Ting

Comment 14 liting 2023-02-01 07:52:53 UTC

(In reply to David Marchand from comment #11)
> - I logged in the vm, and started testpmd, every rdma ioctl was *super* slow
> (I could see this via strace),
>   but ended up initialising fine,
> 
> - after rebooting the vm, I can see in the vm dmesg that the infiniband
> kernel driver is *super* slow to initialise too:
> [    6.615048] mlx5_core 0000:03:00.0: firmware version: 16.31.1014
> [    6.848867] mlx5_core 0000:03:00.0: Rate limit: 127 rates are supported,
> range: 0Mbps to 97656Mbps
> [    7.159308] mlx5_core 0000:04:00.0: firmware version: 16.31.1014
> [    7.352350] mlx5_core 0000:04:00.0: Rate limit: 127 rates are supported,
> range: 0Mbps to 97656Mbps
> [    9.681689] mlx5_core 0000:03:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048)
> RxCqeCmprss(0)
> [   90.681654] mlx5_core 0000:03:00.0: Supported tc offload range - chains:
> 1, prios: 1
> [   98.682242] mlx5_core 0000:04:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048)
> RxCqeCmprss(0)
> [  162.185663] mlx5_core 0000:04:00.0: Supported tc offload range - chains:
> 1, prios: 1
>   After waiting 5 minutes that the kernel driver finished, I started testpmd
> which was super slow, but ended up initialising fine,
> 
> - I stopped the vm, the VF devices were returned to mlx5 kernel driver on
> the host.
>   I started testpmd on those VF in the host, no issue, initialisation seems
> normal, no slowdown,
> 
> - I restarted the vm, kernel was ok, then started testpmd fine, no slowdown,
> 
> - I rebooted the vm, and we are back with the same phenomenon:
> [    6.685098] mlx5_core 0000:03:00.0: firmware version: 16.31.1014
> [    6.889649] mlx5_core 0000:03:00.0: Rate limit: 127 rates are supported,
> range: 0Mbps to 97656Mbps
> [    7.200112] mlx5_core 0000:04:00.0: firmware version: 16.31.1014
> [    7.420885] mlx5_core 0000:04:00.0: Rate limit: 127 rates are supported,
> range: 0Mbps to 97656Mbps
> [    9.685476] mlx5_core 0000:03:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048)
> RxCqeCmprss(0)
> [  101.189469] mlx5_core 0000:03:00.0: Supported tc offload range - chains:
> 1, prios: 1
> [  110.686708] mlx5_core 0000:04:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048)
> RxCqeCmprss(0)
> [  182.627968] mlx5_core 0000:04:00.0: Supported tc offload range - chains:
> 1, prios: 1
> 
> 
> This slowdown in the guest is strange and likely the source of the reported
> issue.
> 
> qemu logs looks ok to me.
> I recorded perf events on the CPU used by qemu, I can't conclude anything
> from it.
> I see some kvm calls, and some idle task.
> The host system itself seems rather idle.
> 
> 
> I don't think I can help more.
> I recommend contacting qemu/kvm people so they have a look at this system.
> 
> 
> 
> One last thing.
> 
> I noticed that there are VPD warnings on the host:
> [root@dell-per730-56 ~]# dmesg | grep -i VPD
> [    6.106574] pci 0000:03:00.0: [Firmware Bug]: disabling VPD access (can't
> determine size of non-standard VPD format)
> [    6.118532] pci 0000:03:00.0: quirk_blacklist_vpd+0x0/0x22 took 11718
> usecs
> [  536.822487] pci 0000:82:00.0: VPD access failed.  This is likely a
> firmware bug on this device.  Contact the card vendor for a firmware update
> [  536.963396] pci 0000:82:00.1: VPD access failed.  This is likely a
> firmware bug on this device.  Contact the card vendor for a firmware update

I thought it is not the vm issue, because when I changed guest to use rhel9.0 or rhel9.1 image, it still had this issue.
If use the common steps without reboot guest, it has no this issue, 
This issue will occur only after reboot the guest. 
In my automated test script, it will configure grub options and reboot guest make it work, and then start testpmd command. So sriov pvp case run failed in the automation case.

Before reboot, there is enp3s0 and enp4s0 inside guest.
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3546sec preferred_lft 3546sec
    inet6 fe80::2d6e:7321:322b:e064/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:de:ad:01:01:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c246:d56f:3db7:67d/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:de:ad:02:02:02 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1f5f:299e:f963:8791/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
       
After reboot, there is no enp3s0 and enp4s0 inside guest. If you can run testpmd successfully, it because the two ports still exist. You can try reboot several times.
[root@localhost ~]# reboot
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:01:02:03 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.123/24 brd 192.168.122.255 scope global dynamic noprefixroute enp2s0
       valid_lft 3579sec preferred_lft 3579sec
    inet6 fe80::3f78:cc40:6622:60af/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
[root@localhost ~]# 
[root@localhost ~]# driverctl -v list-overrides
driverctl: No overridable devices found. Kernel too old?

I think it maybe the kernel issue, my host is rhel9.2 kernel 5.14.0-197.el9.x86_64. When I run the same cases with host using rhel9.1, it has no this issue. 
[root@dell-per730-56 perf]# uname -r
5.14.0-197.el9.x86_64

Comment 15 David Marchand 2023-02-01 07:58:47 UTC

I have nothing to add to my analysis.
Please look at Jason request, thanks.

Comment 16 liting 2023-02-01 11:59:18 UTC

I update the firmware to the latest this afternoon, and rerun the case, it still has this issue.
[root@dell-per730-56 ~]# ethtool -i enp4s0f0np0
driver: mlx5_core
version: 5.14.0-197.el9.x86_64
firmware-version: 16.35.1012 (MT_0000000012)
expansion-rom-version: 
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Job:
https://beaker.engineering.redhat.com/jobs/7487448

You can access to my system to debug. Thanks
dell-per730-56.rhts.eng.pek2.redhat.com(root/redhat)

Comment 17 liting 2023-02-13 03:08:14 UTC

dpdk-22.11-1.el9.x86_64 run on rhel9.2 also exist this issue
https://beaker.engineering.redhat.com/jobs/7524228

Comment 18 liting 2023-04-27 07:18:56 UTC

For bf2 card, it also has this issue.
https://beaker.engineering.redhat.com/jobs/7789400