This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
Bug 2151555 - [virtio-iommu] "VFIO_MAP_DMA failed: File exists" errors when VFIO devices are in the same guest iommu group
Summary: [virtio-iommu] "VFIO_MAP_DMA failed: File exists" errors when VFIO devices ar...
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.2
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Virtualization Maintenance
QA Contact: jinl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-12-07 13:53 UTC by Eric Auger
Modified: 2023-08-07 14:30 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-07 14:29:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker   RHEL-1061 0 None None None 2023-08-07 14:29:54 UTC
Red Hat Issue Tracker RHELPLAN-141547 0 None None None 2022-12-07 13:56:37 UTC

Description Eric Auger 2022-12-07 13:53:30 UTC 
With the setup described in BZ https://bugzilla.redhat.com/show_bug.cgi?id=2103649

where the 2 assigned PFs end up in the same iommu group on the guest (by mistake in that case due to the bug, but this can actually happen if the 2 assigned devices are pluged downstream to a PCIe to PCI bridge), we get spurious qemu errors:

2022-12-07T13:47:03.661326Z qemu-system-x86_64: VFIO_MAP_DMA failed: File exists
2022-12-07T13:47:03.661333Z qemu-system-x86_64: vfio_dma_map(0x556cc5e0acf0, 0xbfbd1000, 0x1000, 0x7f1692853000) = -17 (File exists)

This is due to a bug in the QEMU virtio-iommu device where the replay and remap function only do MAP without prior UNMAP, causing this attempt dma map onto an existing mapping.
Comment 1 Eric Auger 2022-12-07 13:58:23 UTC 
Sent "[PATCH for 8.0 0/2] virtio-iommu: Fix Replay" upstream
Comment 2 jinl 2022-12-13 03:52:35 UTC 
try to reproduce this bz with assigning 2 devices to PCIe to PCI bridge

qemu command
-device '{"driver":"virtio-iommu","id":"iommu0","bus":"pcie.0","addr":"0x4"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16","addr":"0x2"}' \
-device '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16","addr":"0x1"}' \
...

check iommu group in vm
#lspci
01:00.0 PCI bridge: Red Hat, Inc. Device 000e
02:01.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
02:02.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe

#dmesg | grep iommu
[    2.080581] xhci_hcd 0000:04:00.0: Adding to iommu group 0
[    2.917698] pcieport 0000:00:02.0: Adding to iommu group 1
[    2.927124] pcieport 0000:00:02.1: Adding to iommu group 2
[    2.936803] pcieport 0000:00:02.2: Adding to iommu group 3
[    2.945757] pcieport 0000:00:02.3: Adding to iommu group 4
[    2.954896] pcieport 0000:00:02.4: Adding to iommu group 5
[    2.963906] pcieport 0000:00:02.5: Adding to iommu group 6
[    2.972958] pcieport 0000:00:02.6: Adding to iommu group 7
[    2.981013] pcieport 0000:00:02.7: Adding to iommu group 8
[    2.989058] pcieport 0000:00:03.0: Adding to iommu group 9
[    2.998632] pcieport 0000:00:03.1: Adding to iommu group 10
[    3.006949] pcieport 0000:00:03.2: Adding to iommu group 11
[    3.015260] pcieport 0000:00:03.3: Adding to iommu group 12
[    3.024007] pcieport 0000:00:03.4: Adding to iommu group 13
[    3.031627] pcieport 0000:00:03.5: Adding to iommu group 14
[    3.039167] pcieport 0000:00:03.6: Adding to iommu group 15
[    3.057450] virtio-pci 0000:00:01.0: Adding to iommu group 16
[    3.061114] virtio-pci 0000:03:00.0: Adding to iommu group 17
[    3.064687] virtio-pci 0000:05:00.0: Adding to iommu group 18
[    3.068482] virtio-pci 0000:06:00.0: Adding to iommu group 19
[    3.072032] virtio-pci 0000:08:00.0: Adding to iommu group 20
[    3.076255] virtio-pci 0000:09:00.0: Adding to iommu group 21
[    4.110884] ahci 0000:00:1f.2: Adding to iommu group 22
[    6.740302] lpc_ich 0000:00:1f.0: Adding to iommu group 22
[    6.794388] i801_smbus 0000:00:1f.3: Adding to iommu group 22

The assigned devices and pcie-pci-bridge didn't in iommu group.But with intel_iommu, it can be added to the same group.

with intel_iommu

-device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"device-iotlb":true}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16","addr":"0x2"}' \
...

#lspci
01:00.0 PCI bridge: Red Hat, Inc. Device 000e
02:01.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:02.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)

[    2.428317] pci 0000:01:00.0: Adding to iommu group 25
[    2.429055] pci 0000:02:01.0: Adding to iommu group 25
[    2.429800] pci 0000:02:02.0: Adding to iommu group 25

Eric, can you help to check the behavior? Thanks.
Comment 3 Eric Auger 2022-12-21 16:29:03 UTC 
(In reply to jinl from comment #2)
> try to reproduce this bz with assigning 2 devices to PCIe to PCI bridge
> 
> qemu command
> -device
> '{"driver":"virtio-iommu","id":"iommu0","bus":"pcie.0","addr":"0x4"}' \
> -device
> '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
> -device
> '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16",
> "addr":"0x2"}' \
> -device
> '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16",
> "addr":"0x1"}' \
> ...
> 
> check iommu group in vm
> #lspci
> 01:00.0 PCI bridge: Red Hat, Inc. Device 000e
> 02:01.0 USB controller: Intel Corporation C610/X99 series chipset USB
> Enhanced Host Controller #2 (rev 05)
> 02:02.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme
> BCM5720 Gigabit Ethernet PCIe
> 
> #dmesg | grep iommu
> [    2.080581] xhci_hcd 0000:04:00.0: Adding to iommu group 0
> [    2.917698] pcieport 0000:00:02.0: Adding to iommu group 1
> [    2.927124] pcieport 0000:00:02.1: Adding to iommu group 2
> [    2.936803] pcieport 0000:00:02.2: Adding to iommu group 3
> [    2.945757] pcieport 0000:00:02.3: Adding to iommu group 4
> [    2.954896] pcieport 0000:00:02.4: Adding to iommu group 5
> [    2.963906] pcieport 0000:00:02.5: Adding to iommu group 6
> [    2.972958] pcieport 0000:00:02.6: Adding to iommu group 7
> [    2.981013] pcieport 0000:00:02.7: Adding to iommu group 8
> [    2.989058] pcieport 0000:00:03.0: Adding to iommu group 9
> [    2.998632] pcieport 0000:00:03.1: Adding to iommu group 10
> [    3.006949] pcieport 0000:00:03.2: Adding to iommu group 11
> [    3.015260] pcieport 0000:00:03.3: Adding to iommu group 12
> [    3.024007] pcieport 0000:00:03.4: Adding to iommu group 13
> [    3.031627] pcieport 0000:00:03.5: Adding to iommu group 14
> [    3.039167] pcieport 0000:00:03.6: Adding to iommu group 15
> [    3.057450] virtio-pci 0000:00:01.0: Adding to iommu group 16
> [    3.061114] virtio-pci 0000:03:00.0: Adding to iommu group 17
> [    3.064687] virtio-pci 0000:05:00.0: Adding to iommu group 18
> [    3.068482] virtio-pci 0000:06:00.0: Adding to iommu group 19
> [    3.072032] virtio-pci 0000:08:00.0: Adding to iommu group 20
> [    3.076255] virtio-pci 0000:09:00.0: Adding to iommu group 21
> [    4.110884] ahci 0000:00:1f.2: Adding to iommu group 22
> [    6.740302] lpc_ich 0000:00:1f.0: Adding to iommu group 22
> [    6.794388] i801_smbus 0000:00:1f.3: Adding to iommu group 22
> 
> The assigned devices and pcie-pci-bridge didn't in iommu group.But with
> intel_iommu, it can be added to the same group.
This does not look normal. Would it be possible to get access to the machine?

Thanks

Eric
Comment 4 jinl 2022-12-22 01:46:05 UTC 
(In reply to Eric Auger from comment #3)
> This does not look normal. Would it be possible to get access to the machine?
> 
> Thanks
> 
> Eric

Hi Eric,

Host IP: 10.73.224.223
Host passwd: kvmautotest
Guest name: rhel
Guest passwd: redhat

I also install a vm with intel_iommu named intel, you can compare the behavior, the passwd is the same.

Thanks.
Comment 5 Eric Auger 2022-12-22 16:59:41 UTC 
Thank you for the machine coordinates. Looking at it ...
Comment 6 Eric Auger 2022-12-22 18:15:06 UTC 
Hello,

I can reproduce on my ARM machine. I don't want to prevent you from using your machine. I will try to fix on mine. Nevertheless please keep your setup in mind as I may need it in short term to handle the primary BZ topic (-EEXIST replay issue). I will create a separate BZ for the group issue, against qemu.

Thanks

Eric
Comment 7 jinl 2022-12-23 02:18:58 UTC 
(In reply to Eric Auger from comment #6)
> Hello,
> 
> I can reproduce on my ARM machine. I don't want to prevent you from using
> your machine. I will try to fix on mine. Nevertheless please keep your setup
> in mind as I may need it in short term to handle the primary BZ topic
> (-EEXIST replay issue). I will create a separate BZ for the group issue,
> against qemu.
> 
> Thanks
> 
> Eric

File a Bug 2155954 to track the iommu group issue. Thanks.
Comment 8 Eric Auger 2023-02-02 17:17:54 UTC 
Moving the ITR to 9.3 since the fix is not upstream yet and I have difficulties to set up a reproducer (pcie to pci bridge not working with virtio-iommu)
Comment 9 John Ferlan 2023-07-18 12:23:21 UTC 
Eric - Any update on this - is there an upstream fix or should/can we just move this to the backlog until there is
Comment 10 Eric Auger 2023-07-18 18:26:37 UTC 
(In reply to John Ferlan from comment #9)
> Eric - Any update on this - is there an upstream fix or should/can we just
> move this to the backlog until there is
Hi John, yes this is rather low priority at the moment. I reset the ITR and the assignee. Hope this is sufficient.
Note You need to log in before you can comment on or make changes to this bug.