Bug 1811863
| Summary: | Boot guest with device assignment, choose PC machine and "pci=nomsi" in kernel line, guest kernel will show "Call Trace" when booting testpmd(rhel8.2) | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Pei Zhang <pezhang> | |
| Component: | qemu-kvm | Assignee: | Laurent Vivier <lvivier> | |
| qemu-kvm sub component: | Networking | QA Contact: | Pei Zhang <pezhang> | |
| Status: | CLOSED WONTFIX | Docs Contact: | ||
| Severity: | low | |||
| Priority: | low | CC: | aadam, alex.williamson, chayang, jinzhao, juzhang, peterx, virt-maint, yama | |
| Version: | 8.2 | Keywords: | Triaged | |
| Target Milestone: | rc | |||
| Target Release: | 8.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1809978 | |||
| : | 1811885 (view as bug list) | Environment: | ||
| Last Closed: | 2021-01-20 10:08:23 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1809978 | |||
| Bug Blocks: | 1811885 | |||
|
Description
Pei Zhang
2020-03-10 00:50:50 UTC
> -device vfio-pci,host=0000:5e:00.0,id=hostdev0 \ > -device vfio-pci,host=0000:5e:00.1,id=hostdev1 \ Which PCI card are plugged in these ports on the host? > [ 127.889733] irq 10: nobody cared (try booting with the "irqpoll" option) This means kernel has received an interrupt it doesn't manage. There are some related stuffs in v5.1 kernel, but they already merged in kernel-4.18.0-147.9: e8303bb7a75c ("PCI/LINK: Report degraded links via link bandwidth notification") 3e82a7f9031f ("PCI/LINK: Supply IRQ handler so level-triggered IRQs are acked") 15d2aba7c602 ("PCI/portdrv: Use shared MSI/MSI-X vector for Bandwidth Management") 2078e1e7f7e0 ("[pci] PCI/LINK: Add Kconfig option (default off)") Alex, as it's seems related to VFIO, do you have any idea why we have "irq 10: nobody cared" with "pci=nomsi" parameter? I think this is probably a combination of things, including vfio-pci advertising MSI/X interrupts that it cannot configure when used with pci=nomsi and perhaps the dpdk driver ignoring the error from MSI/X setup and allowing the device to continue to generate interrupts. Perhaps there's something we can harden in either vfio-pci or the dpdk drivers to prevent this, but this is pretty low priority issue given that the VM (or host - this is reproducible with DPDK on bare metal) need to be configured with pci=nomsi, which is not a configuration we'd advise for anything other than conducting specific tests. (In reply to Laurent Vivier from comment #1) > > -device vfio-pci,host=0000:5e:00.0,id=hostdev0 \ > > -device vfio-pci,host=0000:5e:00.1,id=hostdev1 \ > > Which PCI card are plugged in these ports on the host? > Hi Laurent, It's X540-AT2(10G, ixgbe) cards. # lspci | grep Eth 5e:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) 5e:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01) Tested on bare metal with: kernel-4.18.0-193.el8.x86_64 dpdk-19.11-4.el8.x86_64 # dpdk-devbind --status Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe 0000:05:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' drv=vfio-pci unused=ixgbe When I run testpmd directly on a host started with pci=nomsi testpmd correctly reports MSI-X errors: # /usr/bin/testpmd -l 1,2,3 -n 4 -d /usr/lib64/librte_pmd_ixgbe.so.20.0 -w 0000:05:00.0 -w 0000:05:00.1 -- --nb-cores=2 -i --disable-rss --rxd=512 --txd=512 --rxq=1 --txq=1 EAL: Detected 16 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free hugepages reported in hugepages-2048kB EAL: No free hugepages reported in hugepages-2048kB EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:05:00.0 on NUMA socket 0 EAL: probe driver: 8086:1528 net_ixgbe EAL: using IOMMU type 1 (Type 1) EAL: Error enabling MSI-X interrupts for fd 39 EAL: PCI device 0000:05:00.1 on NUMA socket 0 EAL: probe driver: 8086:1528 net_ixgbe EAL: Error enabling MSI-X interrupts for fd 44 Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=163456, size=2176, socket=1 testpmd: preferred mempool ops selected: ring_mp_mc testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) EAL: Error disabling MSI-X interrupts for fd 39 EAL: Error enabling MSI-X interrupts for fd 39 Port 0: A0:36:9F:65:88:74 Configuring Port 1 (socket 0) EAL: Error disabling MSI-X interrupts for fd 44 EAL: Error enabling MSI-X interrupts for fd 44 Port 1: A0:36:9F:65:88:76 Checking link statuses... Done testpmd> quit Stopping port 0... Stopping ports... Done Stopping port 1... Stopping ports... Done Shutting down port 0... Closing ports... EAL: Error disabling MSI-X interrupts for fd 39 Done Shutting down port 1... Closing ports... EAL: Error disabling MSI-X interrupts for fd 44 Done Bye... I've started a guest with the command line of comment #0 with pci=nomsi for the _guest_kernel and I have exactly the same error: # /usr/bin/testpmd -l 1,2,3 -n 4 -d /usr/lib64/librte_pmd_ixgbe.so.20.0 -w 0000:00:05.0 -w 0000:00:06.0 -- --nb-cores=2 -i --disable-rss --rxd=512 --txd=512 --rxq=1 --txq=1 EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free hugepages reported in hugepages-2048kB EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:00:05.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:1528 net_ixgbe EAL: using IOMMU type 8 (No-IOMMU) EAL: Error enabling MSI-X interrupts for fd 31 EAL: PCI device 0000:00:06.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:1528 net_ixgbe EAL: Error enabling MSI-X interrupts for fd 35 Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) EAL: Error disabling MSI-X interrupts for fd 31 EAL: Error enabling MSI-X interrupts for fd 31 Port 0: A0:36:9F:65:88:74 Configuring Port 1 (socket 0) EAL: Error disabling MSI-X interrupts for fd 35 EAL: Error enabling MSI-X interrupts for fd 35 Port 1: A0:36:9F:65:88:76 Checking link statuses... Done testpmd> Stopping port 0... Stopping ports... Done Stopping port 1... Stopping ports... Done Shutting down port 0... Closing ports... EAL: Error disabling MSI-X interrupts for fd 31 Done Shutting down port 1... Closing ports... EAL: Error disabling MSI-X interrupts for fd 35 Done Bye... Laurent, in order to see the original issue I believe you'll need to be on a system where the legacy IRQ for the device is shared with other devices. This will leave the APIC through which the device would signal an INTx enabled, thus if the device generates interrupts after MSI-X configuration fails, it will do so via the INTx pin of the device, where no handler is installed. I think the resolution is that vfio-pci needs to have a backup IRQ handler installed for this situation to actively mask the device when it generates an interrupt with no user configured consumer. Thanks If I start guest without pci=nomsi for guest kernel but with pci=nomsi for host kernel I have some warnings reported by qemu: qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to modify vector, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to modify vector, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 qemu-kvm: vfio: failed to enable vectors, -1 But I can start testpmd without error: # /usr/bin/testpmd -l 1,2,3 -n 4 -d /usr/lib64/librte_pmd_ixgbe.so.20.0 -w 0000:00:05.0 -w 0000:00:06.0 -- --nb-cores=2 -i --disable-rss --rxd=512 --txd=512 --rxq=1 --txq=1 EAL: Detected 4 lcore(s) EAL: Detected 1 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: No free hugepages reported in hugepages-2048kB EAL: No available hugepages reported in hugepages-2048kB EAL: Probing VFIO support... EAL: VFIO support initialized EAL: PCI device 0000:00:05.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:1528 net_ixgbe EAL: using IOMMU type 8 (No-IOMMU) EAL: PCI device 0000:00:06.0 on NUMA socket -1 EAL: Invalid NUMA socket, default to 0 EAL: probe driver: 8086:1528 net_ixgbe Interactive-mode selected testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=163456, size=2176, socket=0 testpmd: preferred mempool ops selected: ring_mp_mc Configuring Port 0 (socket 0) Port 0: A0:36:9F:65:88:74 Configuring Port 1 (socket 0 Port 1: A0:36:9F:65:88:76 Checking link statuses... Done testpmd> And I have no error in guest and host kernel dmesg (In reply to Alex Williamson from comment #7) > Laurent, in order to see the original issue I believe you'll need to be on a > system where the legacy IRQ for the device is shared with other devices. > This will leave the APIC through which the device would signal an INTx > enabled, thus if the device generates interrupts after MSI-X configuration > fails, it will do so via the INTx pin of the device, where no handler is > installed. I think the resolution is that vfio-pci needs to have a backup > IRQ handler installed for this situation to actively mask the device when it > generates an interrupt with no user configured consumer. Thanks Thank you Alex. How can I identify such devices? I also wanted to check if dpdk correctly manages systems with MSI/X disabled and it seems ok (according to errors in comment #6) Low priority issue, moving to 8.4.0 Close this bz WONTFIX as discussed in the internal virtio-networking sync meeting. This is a low priority bz and it's corner case. |