Red Hat Bugzilla – Bug 1235581
RFE: Enable the intel-iommu device in QEMU
Last modified: 2017-04-12 05:56:17 EDT
Requesting representation of iommu device support in libvirt XML. +++ This bug was initially created as a clone of Bug #1235580 +++ Description of problem: The vast majority of testing by both the community on OpenStack upstream and Red Hat QE for RHEL-OSP products is done inside virtualized guests, using either TCG or nested-KVM. One of the few areas where we are unable to use virtualization is in the area of PCI device assignment. This means we are limited in our ability to setup automated test suites for OpenStack, by access to small pool of physical hardware nodes. If QEMU were to provide IOMMU device emulation then it would be possible to move some of our PCI device assignment testing into a virtualized environment. The new QEMU intel-iommu device looks like it would satisfy the needs for basic testing, even if there are a limited number of QEMU PCI devices we can use it with. NB, this is not a request for intel-iommu to be marked fully supported for production deployment. RHEL OpenStack team would merely like it for dev/test scenarios at this time, so tech-preview would be sufficient enablement, if supportability is a concern. As such there is also no hard deadline we need it by.
Is this command line for q35 machines all that is needed here? -device intel-iommu It seems broken upstream unless I apply the following fix: https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03548.html qemu-git: Option '-device intel-iommu' cannot be handled by this machine
Proposed upstream patches: https://www.redhat.com/archives/libvir-list/2016-June/msg01645.html
v2: https://www.redhat.com/archives/libvir-list/2016-July/msg00334.html
Pushed upstream as: commit ea0ed35d6efcba9e79c76d7b57959f553b76224a Author: Ján Tomko <jtomko@redhat.com> CommitDate: 2016-07-12 12:36:13 +0200 Introduce <iommu> device A device with an attribute 'model', with just one model so far: <devices> ... <iommu model='intel'/> </devices> https://bugzilla.redhat.com/show_bug.cgi?id=1235580 commit 8e7e79738d47d7881eb6ccc61ddcb0cfba418d94 Author: Ján Tomko <jtomko@redhat.com> CommitDate: 2016-07-12 12:36:13 +0200 Add QEMU_CAPS_DEVICE_INTEL_IOMMU Check whether QEMU supports -device intel-iommu Note that the presence of this option does not mean that it's usable because of a bug in earlier QEMU versions, but it's better than nothing. https://bugzilla.redhat.com/show_bug.cgi?id=1235580 commit 4c382376da95bf84c9903e62c714ffa33f85d404 Author: Ján Tomko <jtomko@redhat.com> CommitDate: 2016-07-12 12:36:13 +0200 qemu: format intel-iommu on the command line <devices> <iommu model='intel'/> </devices> results in: -device intel-iommu https://bugzilla.redhat.com/show_bug.cgi?id=1235580 git describe: v2.0.0-126-g4c38237
Hi Ján, I rpmbuild the newest patches, try to test this issue and get error as Comment 4. Is it caused by the qemu-kvm bug https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? # rpm -qa | grep qemu-kvm-rhev qemu-kvm-rhev-2.6.0-11.el7.x86_64 # rpm -q libvirt libvirt-2.1.0-1.el7.x86_64 # virsh dumpxml r7.1 .... <os> <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type> <boot dev='hd'/> </os> .... <controller type='pci' index='0' model='pcie-root'/> .... <iommu model='intel'/> ... # virsh start r7.1 error: Failed to start domain r7.1 error: internal error: process exited while connecting to monitor: 2016-07-18T11:12:20.827417Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine
Hi Ján, Would you please help to look at the error and question in the comment10 ？ Thank you in advanced
(In reply to Jingjing Shao from comment #10) > I rpmbuild the newest patches, try to test this issue and get error as > Comment 4. Is it caused by the qemu-kvm bug > https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? > Bug 1350196 is tracking the switch from -machine,iommu=on to -device.
Hi Ján, I try to verify this bug，but I get result as below. I have three question about the result.Can you help to check it and update something? thank you in advance. I find that in the guest the NIC device share the same IOMMU group as the step 5. Is is right? If so , the device can not be assigned to the nested guest. Or will it be supported in https://bugzilla.redhat.com/show_bug.cgi?id=1283251 ? # rpm -q libvirt libvirt-2.0.0-5.el7.x86_64 # rpm -q qemu-kvm-rhev qemu-kvm-rhev-2.6.0-20.el7.x86_64 1.prepare a guest with q35 machine type as below and add <iommu model='intel'/> # virsh dumpxml q35-js <domain type='kvm' id='12'> <name>q35-js</name> <uuid>251e7c62-3df4-4354-9a5b-0c7972035393</uuid> ..... <os> <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type> - <boot dev='hd'/> </os> ..... <cpu mode='custom' match='exact'> <model fallback='allow'>SandyBridge</model> <feature policy='require' name='vmx'/> </cpu> ... <devices> ... <controller type='pci' index='0' model='pcie-root'/> <interface type='network'> <mac address='54:52:00:54:9e:f4'/> <source network='default' bridge='virbr0'/> <target dev='vnet0'/> <model type='e1000'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </interface> <iommu model='intel'/> </devices> 2. start the guest Already add the "intel_iommu=on" in kernel commend line # virsh start q35-js Domain q35-js started 3.Check the qemu commend line # ps -ef | grep qemu | grep iommu ....-device intel-iommu..... 4.In the guest # dmesg | grep IOMMU [ 0.000000] Intel-IOMMU: enabled [ 0.110758] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap 12008c22260206 ecap f02 [ 0.671621] IOMMU 0 0xfed90000: using Queued invalidation [ 0.671623] IOMMU: Setting RMRR: [ 0.671624] IOMMU: Prepare 0-16MiB unity mapping for LPC [ 0.671658] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] # dmesg | grep DMAR [ 0.000000] ACPI: DMAR 000000003ffe2440 00040 (v01 BOCHS BXPCDMAR 00000001 BXPC 00000001) [ 0.671490] DMAR: No RMRR found [ 0.671493] DMAR: No ATSR found # ll /sys/kernel/iommu_groups/ total 0 drwxr-xr-x. 3 root root 0 Aug 19 20:05 0 drwxr-xr-x. 3 root root 0 Aug 19 20:05 1 drwxr-xr-x. 3 root root 0 Aug 19 20:05 2 drwxr-xr-x. 3 root root 0 Aug 19 20:05 3 drwxr-xr-x. 3 root root 0 Aug 19 20:05 4 drwxr-xr-x. 3 root root 0 Aug 19 20:05 5 drwxr-xr-x. 3 root root 0 Aug 19 20:05 6 drwxr-xr-x. 3 root root 0 Aug 19 20:05 7 5.In the guest # lspci | grep Eth 02:02.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03) # virsh nodedev-list --tree ... | +- pci_0000_00_1e_0 | | | +- pci_0000_01_00_0 | | | +- pci_0000_02_01_0 | +- pci_0000_02_02_0 | | | +- net_enp2s2_54_52_00_54_9e_f4 .... # virsh nodedev-dumpxml pci_0000_02_02_0 <device> <name>pci_0000_02_02_0</name> <path>/sys/devices/pci0000:00/0000:00:1e.0/0000:01:00.0/0000:02:02.0</path> <parent>pci_0000_01_00_0</parent> <driver> <name>e1000</name> </driver> <capability type='pci'> <domain>0</domain> <bus>2</bus> <slot>2</slot> <function>0</function> <product id='0x100e'>82540EM Gigabit Ethernet Controller</product> <vendor id='0x8086'>Intel Corporation</vendor> <iommuGroup number='6'> <address domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </iommuGroup> </capability> </device> 6. In the guest, I start a nested guest # virsh start jishao Domain jishao started # cat vf.xml <hostdev type='pci' managed='yes'> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/> </source> </hostdev> # virsh attach-device jishao vf.xml error: Failed to attach device from vf.xml error: internal error: unable to execute QEMU command 'device_add': Device initialization failed.
One more question， Does <iommu model='intel'/> not support migration ? Why is it a non-migratable device 'iommu-intel' ? I can not find more info in the libvirt.org http://libvirt.org/formatdomain.html#elementsIommu # virsh save r7.1 r7.1.save error: Failed to save domain r7.1 to r7.1.save error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'iommu-intel'
Hi Ján, Would you please help to check the question in comment15 and comment16 and update something? thank you in advance.
According to the qemu bug: https://bugzilla.redhat.com/show_bug.cgi?id=1350196 the step5 in the comment15 is enough for this bug. so I change the status to verified. I will track the issues in the step6 and comment 16 in the rhel7.4
Some testing scenario for supplement 1. add the <iommu model='intel'/> to the guest with q35 machine type # virsh dumpxml rhel7.3 | grep iommu <iommu model='intel'/> # virsh start q35-js Domain q35-js started # ps -ef | grep qemu | grep "iommu" .....-device intel-iommu ....... 2. add the <iommu model='intel'/> to the guest with 'pc-i440fx-rhel7.3.0' machine type # virsh start rhel7.3 error: Failed to start domain rhel7.3 error: unsupported configuration: IOMMU device: 'intel' is only supported with Q35 machines 3. coldplug reject # virsh attach-device q35-js iommu.xml --config error: Failed to attach device from iommu.xml error: Operation not supported: persistent attach of device 'iommu' is not supported hotplug reject # virsh start q35-js Domain q35-js started # virsh attach-device q35-js iommu.xml error: Failed to attach device from iommu.xml error: Operation not supported: live attach of device 'iommu' is not supported 4. Test with lower version of qemu-kvm-rhev # rpm -q qemu-kvm-rhev qemu-kvm-rhev-2.6.0-17.el7.x86_64 # virsh start q35-js error: Failed to start domain q35-js error: internal error: qemu unexpectedly closed the monitor: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5] 2016-09-28T11:29:21.804359Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html