Bug 2151555

Summary:	[virtio-iommu] "VFIO_MAP_DMA failed: File exists" errors when VFIO devices are in the same guest iommu group
Product:	Red Hat Enterprise Linux 9	Reporter:	Eric Auger <eric.auger>
Component:	qemu-kvm	Assignee:	Virtualization Maintenance <virt-maint>
qemu-kvm sub component:	Devices	QA Contact:	jinl
Status:	CLOSED MIGRATED	Docs Contact:
Severity:	medium
Priority:	medium	CC:	alex.williamson, coli, jinl, jinzhao, juzhang, mst, peterx, virt-maint, yama, yanghliu, zhguo
Version:	9.2	Keywords:	MigratedToJIRA, Triaged
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-08-07 14:29:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eric Auger 2022-12-07 13:53:30 UTC

With the setup described in BZ https://bugzilla.redhat.com/show_bug.cgi?id=2103649

where the 2 assigned PFs end up in the same iommu group on the guest (by mistake in that case due to the bug, but this can actually happen if the 2 assigned devices are pluged downstream to a PCIe to PCI bridge), we get spurious qemu errors:

2022-12-07T13:47:03.661326Z qemu-system-x86_64: VFIO_MAP_DMA failed: File exists
2022-12-07T13:47:03.661333Z qemu-system-x86_64: vfio_dma_map(0x556cc5e0acf0, 0xbfbd1000, 0x1000, 0x7f1692853000) = -17 (File exists)

This is due to a bug in the QEMU virtio-iommu device where the replay and remap function only do MAP without prior UNMAP, causing this attempt dma map onto an existing mapping.

Comment 1 Eric Auger 2022-12-07 13:58:23 UTC

Sent "[PATCH for 8.0 0/2] virtio-iommu: Fix Replay" upstream

Comment 2 jinl 2022-12-13 03:52:35 UTC

try to reproduce this bz with assigning 2 devices to PCIe to PCI bridge

qemu command
-device '{"driver":"virtio-iommu","id":"iommu0","bus":"pcie.0","addr":"0x4"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16","addr":"0x2"}' \
-device '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16","addr":"0x1"}' \
...

check iommu group in vm
#lspci
01:00.0 PCI bridge: Red Hat, Inc. Device 000e
02:01.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
02:02.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe

#dmesg | grep iommu
[    2.080581] xhci_hcd 0000:04:00.0: Adding to iommu group 0
[    2.917698] pcieport 0000:00:02.0: Adding to iommu group 1
[    2.927124] pcieport 0000:00:02.1: Adding to iommu group 2
[    2.936803] pcieport 0000:00:02.2: Adding to iommu group 3
[    2.945757] pcieport 0000:00:02.3: Adding to iommu group 4
[    2.954896] pcieport 0000:00:02.4: Adding to iommu group 5
[    2.963906] pcieport 0000:00:02.5: Adding to iommu group 6
[    2.972958] pcieport 0000:00:02.6: Adding to iommu group 7
[    2.981013] pcieport 0000:00:02.7: Adding to iommu group 8
[    2.989058] pcieport 0000:00:03.0: Adding to iommu group 9
[    2.998632] pcieport 0000:00:03.1: Adding to iommu group 10
[    3.006949] pcieport 0000:00:03.2: Adding to iommu group 11
[    3.015260] pcieport 0000:00:03.3: Adding to iommu group 12
[    3.024007] pcieport 0000:00:03.4: Adding to iommu group 13
[    3.031627] pcieport 0000:00:03.5: Adding to iommu group 14
[    3.039167] pcieport 0000:00:03.6: Adding to iommu group 15
[    3.057450] virtio-pci 0000:00:01.0: Adding to iommu group 16
[    3.061114] virtio-pci 0000:03:00.0: Adding to iommu group 17
[    3.064687] virtio-pci 0000:05:00.0: Adding to iommu group 18
[    3.068482] virtio-pci 0000:06:00.0: Adding to iommu group 19
[    3.072032] virtio-pci 0000:08:00.0: Adding to iommu group 20
[    3.076255] virtio-pci 0000:09:00.0: Adding to iommu group 21
[    4.110884] ahci 0000:00:1f.2: Adding to iommu group 22
[    6.740302] lpc_ich 0000:00:1f.0: Adding to iommu group 22
[    6.794388] i801_smbus 0000:00:1f.3: Adding to iommu group 22

The assigned devices and pcie-pci-bridge didn't in iommu group.But with intel_iommu, it can be added to the same group.

with intel_iommu

-device '{"driver":"intel-iommu","id":"iommu0","intremap":"on","caching-mode":true,"device-iotlb":true}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16","addr":"0x2"}' \
...

#lspci
01:00.0 PCI bridge: Red Hat, Inc. Device 000e
02:01.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:02.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)

[    2.428317] pci 0000:01:00.0: Adding to iommu group 25
[    2.429055] pci 0000:02:01.0: Adding to iommu group 25
[    2.429800] pci 0000:02:02.0: Adding to iommu group 25

Eric, can you help to check the behavior? Thanks.

Comment 3 Eric Auger 2022-12-21 16:29:03 UTC

(In reply to jinl from comment #2)
> try to reproduce this bz with assigning 2 devices to PCIe to PCI bridge
> 
> qemu command
> -device
> '{"driver":"virtio-iommu","id":"iommu0","bus":"pcie.0","addr":"0x4"}' \
> -device
> '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.1","addr":"0x0"}' \
> -device
> '{"driver":"vfio-pci","host":"0000:02:00.0","id":"hostdev0","bus":"pci.16",
> "addr":"0x2"}' \
> -device
> '{"driver":"vfio-pci","host":"0000:00:1a.0","id":"hostdev1","bus":"pci.16",
> "addr":"0x1"}' \
> ...
> 
> check iommu group in vm
> #lspci
> 01:00.0 PCI bridge: Red Hat, Inc. Device 000e
> 02:01.0 USB controller: Intel Corporation C610/X99 series chipset USB
> Enhanced Host Controller #2 (rev 05)
> 02:02.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme
> BCM5720 Gigabit Ethernet PCIe
> 
> #dmesg | grep iommu
> [    2.080581] xhci_hcd 0000:04:00.0: Adding to iommu group 0
> [    2.917698] pcieport 0000:00:02.0: Adding to iommu group 1
> [    2.927124] pcieport 0000:00:02.1: Adding to iommu group 2
> [    2.936803] pcieport 0000:00:02.2: Adding to iommu group 3
> [    2.945757] pcieport 0000:00:02.3: Adding to iommu group 4
> [    2.954896] pcieport 0000:00:02.4: Adding to iommu group 5
> [    2.963906] pcieport 0000:00:02.5: Adding to iommu group 6
> [    2.972958] pcieport 0000:00:02.6: Adding to iommu group 7
> [    2.981013] pcieport 0000:00:02.7: Adding to iommu group 8
> [    2.989058] pcieport 0000:00:03.0: Adding to iommu group 9
> [    2.998632] pcieport 0000:00:03.1: Adding to iommu group 10
> [    3.006949] pcieport 0000:00:03.2: Adding to iommu group 11
> [    3.015260] pcieport 0000:00:03.3: Adding to iommu group 12
> [    3.024007] pcieport 0000:00:03.4: Adding to iommu group 13
> [    3.031627] pcieport 0000:00:03.5: Adding to iommu group 14
> [    3.039167] pcieport 0000:00:03.6: Adding to iommu group 15
> [    3.057450] virtio-pci 0000:00:01.0: Adding to iommu group 16
> [    3.061114] virtio-pci 0000:03:00.0: Adding to iommu group 17
> [    3.064687] virtio-pci 0000:05:00.0: Adding to iommu group 18
> [    3.068482] virtio-pci 0000:06:00.0: Adding to iommu group 19
> [    3.072032] virtio-pci 0000:08:00.0: Adding to iommu group 20
> [    3.076255] virtio-pci 0000:09:00.0: Adding to iommu group 21
> [    4.110884] ahci 0000:00:1f.2: Adding to iommu group 22
> [    6.740302] lpc_ich 0000:00:1f.0: Adding to iommu group 22
> [    6.794388] i801_smbus 0000:00:1f.3: Adding to iommu group 22
> 
> The assigned devices and pcie-pci-bridge didn't in iommu group.But with
> intel_iommu, it can be added to the same group.
This does not look normal. Would it be possible to get access to the machine?

Thanks

Eric

Comment 4 jinl 2022-12-22 01:46:05 UTC

(In reply to Eric Auger from comment #3)
> This does not look normal. Would it be possible to get access to the machine?
> 
> Thanks
> 
> Eric

Hi Eric,

Host IP: 10.73.224.223
Host passwd: kvmautotest
Guest name: rhel
Guest passwd: redhat

I also install a vm with intel_iommu named intel, you can compare the behavior, the passwd is the same.

Thanks.

Comment 5 Eric Auger 2022-12-22 16:59:41 UTC

Thank you for the machine coordinates. Looking at it ...

Comment 6 Eric Auger 2022-12-22 18:15:06 UTC

Hello,

I can reproduce on my ARM machine. I don't want to prevent you from using your machine. I will try to fix on mine. Nevertheless please keep your setup in mind as I may need it in short term to handle the primary BZ topic (-EEXIST replay issue). I will create a separate BZ for the group issue, against qemu.

Thanks

Eric

Comment 7 jinl 2022-12-23 02:18:58 UTC

(In reply to Eric Auger from comment #6)
> Hello,
> 
> I can reproduce on my ARM machine. I don't want to prevent you from using
> your machine. I will try to fix on mine. Nevertheless please keep your setup
> in mind as I may need it in short term to handle the primary BZ topic
> (-EEXIST replay issue). I will create a separate BZ for the group issue,
> against qemu.
> 
> Thanks
> 
> Eric

File a Bug 2155954 to track the iommu group issue. Thanks.

Comment 8 Eric Auger 2023-02-02 17:17:54 UTC

Moving the ITR to 9.3 since the fix is not upstream yet and I have difficulties to set up a reproducer (pcie to pci bridge not working with virtio-iommu)

Comment 9 John Ferlan 2023-07-18 12:23:21 UTC

Eric - Any update on this - is there an upstream fix or should/can we just move this to the backlog until there is

Comment 10 Eric Auger 2023-07-18 18:26:37 UTC

(In reply to John Ferlan from comment #9)
> Eric - Any update on this - is there an upstream fix or should/can we just
> move this to the backlog until there is
Hi John, yes this is rather low priority at the moment. I reset the ITR and the assignee. Hope this is sufficient.