Bug 1738450

Summary:	For intel-iommu, qemu shows conflict behaviors between booting a guest with vfio and hot plugging vfio device
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	Pei Zhang <pezhang>
Component:	qemu-kvm	Assignee:	Peter Xu <peterx>
qemu-kvm sub component:	General	QA Contact:	Pei Zhang <pezhang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	alex.williamson, bdas, chayang, ddepaula, jinzhao, juzhang, knoel, peterx, rbalakri, virt-maint, yanghliu
Version:	8.1
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1738440	Environment:
Last Closed:	2020-05-05 09:47:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1738440
Bug Blocks:

Description Pei Zhang 2019-08-07 08:16:39 UTC

+++ This bug was initially created as a clone of Bug #1738440 +++

Description of problem:

(1)Booting guest with device assignment and vIOMMU without "caching-mode=on", qemu can work well. 

(2)However when booting guest with vIOMMU without "caching-mode=on", then hot plug a vfio device, qemu will quit.

So qemu shows conflict behaviors. If caching-mode=on is a must for intel-iommu to enable device assignment with IOMMU protection, qemu should reject to boot in scenario (1). If caching-mode=on is not a must, qemu should allow hot plug a vfio device.


Version-Release number of selected component (if applicable):
4.18.0-128.el8.x86_64
qemu-kvm-2.12.0-83.module+el8.1.0+3852+0ba8aef0.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Boot qemu with vIOMMU without "caching-mode=on", full cmd refer to [1]

-device intel-iommu,intremap=on,device-iotlb=on \

2. Hot plug a vfio device, qemu quit.

(qemu) device_add vfio-pci,host=0000:5e:00.0,bus=root.3 
We need to set caching-mode=1 for intel-iommu to enable device assignment with IOMMU protection.
# 

3. Boot qemu with vfio device and vIOMMU without "caching-mode=on", qemu works well.

-device intel-iommu,intremap=on,device-iotlb=on \
-device vfio-pci,host=0000:5e:00.0,bus=root.3 \


Actual results:
qemu shows conflict behaviors when boot/hot-plug vfio with vIOMMU but without caching-mode=on.

Expected results:
qemu should show consistent behavior when boot/hot-plug vfio with vIOMMU but without caching-mode=on.

Additional info:
1. This bug has been closed in rhel7.
Bug 1441605 - qemu crash when attach a hostdev device to the guest with intel-iommu device enabled

2. Bug 1622209 - libvirt should notify the user if caching-mode isn't specified for iommu emulation in certain cases 


Reference:
[1]
/usr/libexec/qemu-kvm \
-name rhel8.1 \
-M q35,kernel-irqchip=split \
-cpu Haswell-noTSX \
-m 3G \
-device intel-iommu,intremap=on,device-iotlb=on \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,chassis=1 \
-device pcie-root-port,id=root.2,chassis=2 \
-device pcie-root-port,id=root.3,chassis=3 \
-blockdev driver=file,cache.direct=off,cache.no-flush=on,filename=/home/images_nfv-virt-rt-kvm//rhel8.0.qcow2,node-name=my_file \
-blockdev driver=qcow2,node-name=my,file=my_file \
-device virtio-blk-pci,drive=my,id=virtio-blk0,bus=root.1 \
-vnc :1 \
-monitor stdio \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=88:66:da:5f:dd:01,bus=root.2 \

Comment 1 Pei Zhang 2019-08-07 08:18:52 UTC

Versions:
4.18.0-128.el8.x86_64
qemu v4.1.0-rc3

Comment 6 Pei Zhang 2019-12-06 09:16:31 UTC

Verified this bz with qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64:

Steps following Description.

After step 2, qemu prompts warning info and works well. This is expected behavior.

(qemu) device_add vfio-pci,host=0000:5e:00.0,bus=root.3 
Error: Device assignment is not allowed without enabling caching-mode=on for Intel IOMMU.


After step 3, qemu fails reboot without "caching-mode=on". This is expected behavior.

-device intel-iommu,intremap=on,device-iotlb=on \
-device vfio-pci,host=0000:5e:00.0,bus=root.3 \

(qemu) qemu-kvm: -device vfio-pci,host=0000:5e:00.0,bus=root.3: warning: vfio 0000:5e:00.0: failed to setup resample irqfd: Invalid argument
qemu-kvm: We need to set caching-mode=on for intel-iommu to enable device assignment with IOMMU protection.


So this bug has been fixed very well. Move to 'Verified'.

Comment 7 Ademar Reis 2020-02-05 23:02:11 UTC

QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Comment 9 errata-xmlrpc 2020-05-05 09:47:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017