Bug 1235581 - RFE: Enable the intel-iommu device in QEMU
Summary: RFE: Enable the intel-iommu device in QEMU
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
medium
unspecified
Target Milestone: rc
: ---
Assignee: Ján Tomko
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1235580 1350196
Blocks: 1283104 1288337
TreeView+ depends on / blocked
 
Reported: 2015-06-25 08:56 UTC by Daniel Berrangé
Modified: 2017-04-12 09:56 UTC (History)
15 users (show)

Fixed In Version: libvirt-2.0.0-4.el7
Doc Type: Enhancement
Doc Text:
Clone Of: 1235580
Environment:
Last Closed: 2016-11-03 18:19:17 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1433994 None CLOSED libvirt attempts migration of a domain with intel-iommu device 2019-09-18 02:23:09 UTC
Red Hat Product Errata RHSA-2016:2577 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Internal Links: 1433994

Description Daniel Berrangé 2015-06-25 08:56:09 UTC
Requesting representation of iommu device support in libvirt XML.

+++ This bug was initially created as a clone of Bug #1235580 +++

Description of problem:
The vast majority of testing by both the community on OpenStack upstream and Red Hat QE for RHEL-OSP products is done inside virtualized guests, using either TCG or nested-KVM.

One of the few areas where we are unable to use virtualization is in the area of PCI device assignment. This means we are limited in our ability to setup automated test suites for OpenStack, by access to small pool of physical hardware nodes. If QEMU were to provide IOMMU device emulation then it would be possible to move some of our PCI device assignment testing into a virtualized environment.

The new QEMU intel-iommu device looks like it would satisfy the needs for basic testing, even if there are a limited number of QEMU PCI devices we can use it with.

NB, this is not a request for intel-iommu to be marked fully supported for production deployment. RHEL OpenStack team would merely like it for dev/test scenarios at this time, so tech-preview would be sufficient enablement, if supportability is a concern. As such there is also no hard deadline we need it by.

Comment 4 Ján Tomko 2016-06-23 08:31:26 UTC
Is this command line for q35 machines all that is needed here?
-device intel-iommu

It seems broken upstream unless I apply the following fix:
https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03548.html

qemu-git: Option '-device intel-iommu' cannot be handled by this machine

Comment 7 Ján Tomko 2016-06-23 10:47:37 UTC
Proposed upstream patches:
https://www.redhat.com/archives/libvir-list/2016-June/msg01645.html

Comment 9 Ján Tomko 2016-07-12 10:49:06 UTC
Pushed upstream as:
commit ea0ed35d6efcba9e79c76d7b57959f553b76224a
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-12 12:36:13 +0200

    Introduce <iommu> device
    
    A device with an attribute 'model', with just one model
    so far:
    
    <devices>
      ...
      <iommu model='intel'/>
    </devices>
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

commit 8e7e79738d47d7881eb6ccc61ddcb0cfba418d94
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-12 12:36:13 +0200

    Add QEMU_CAPS_DEVICE_INTEL_IOMMU
    
    Check whether QEMU supports -device intel-iommu
    
    Note that the presence of this option does not mean that it's
    usable because of a bug in earlier QEMU versions, but it's
    better than nothing.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

commit 4c382376da95bf84c9903e62c714ffa33f85d404
Author:     Ján Tomko <jtomko@redhat.com>
CommitDate: 2016-07-12 12:36:13 +0200

    qemu: format intel-iommu on the command line
    
    <devices>
      <iommu model='intel'/>
    </devices>
    
    results in:
    
    -device intel-iommu
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

git describe: v2.0.0-126-g4c38237

Comment 10 Jingjing Shao 2016-07-18 11:22:33 UTC
Hi Ján,

I rpmbuild the newest patches, try to test this issue and get error as Comment 4.  Is it caused by the qemu-kvm bug https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? 

# rpm -qa | grep qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-11.el7.x86_64
# rpm -q libvirt
libvirt-2.1.0-1.el7.x86_64



# virsh dumpxml r7.1
....
<os>
    <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
</os>
....
<controller type='pci' index='0' model='pcie-root'/>
....
<iommu model='intel'/>
...


# virsh start r7.1
error: Failed to start domain r7.1
error: internal error: process exited while connecting to monitor: 2016-07-18T11:12:20.827417Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine

Comment 11 Jingjing Shao 2016-07-18 11:27:38 UTC
Hi Ján,

Would you please help to look at the error and question in the comment10 ? Thank you in advanced

Comment 12 Ján Tomko 2016-07-25 08:29:10 UTC
(In reply to Jingjing Shao from comment #10)
> I rpmbuild the newest patches, try to test this issue and get error as
> Comment 4.  Is it caused by the qemu-kvm bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? 
> 

Bug 1350196 is tracking the switch from -machine,iommu=on to -device.

Comment 15 Jingjing Shao 2016-08-19 12:23:29 UTC
Hi Ján,
I try to verify this bug,but I get result as below. I have three question about the result.Can you help to  check it and update something? thank you in advance.

I find that in the guest the NIC device share the same IOMMU group as the step 5. Is is right?
If so , the device can not be assigned to the nested guest.

Or will it be supported in https://bugzilla.redhat.com/show_bug.cgi?id=1283251 ?

# rpm -q libvirt
libvirt-2.0.0-5.el7.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-20.el7.x86_64

1.prepare a guest with q35 machine type as below and add <iommu model='intel'/>

# virsh dumpxml q35-js
<domain type='kvm' id='12'>
  <name>q35-js</name>
  <uuid>251e7c62-3df4-4354-9a5b-0c7972035393</uuid>
.....
<os>
    <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type>  -
    <boot dev='hd'/>
</os>
.....
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <feature policy='require' name='vmx'/>
  </cpu>
...
<devices>
 ...
<controller type='pci' index='0' model='pcie-root'/>
<interface type='network'>
      <mac address='54:52:00:54:9e:f4'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
</interface>

<iommu model='intel'/>
</devices>


2. start the guest
Already add the "intel_iommu=on" in kernel commend line
# virsh start q35-js
Domain q35-js started

3.Check the qemu commend line 
# ps -ef | grep qemu | grep iommu
....-device intel-iommu.....

4.In the guest
# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.110758] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap 12008c22260206 ecap f02
[    0.671621] IOMMU 0 0xfed90000: using Queued invalidation
[    0.671623] IOMMU: Setting RMRR:
[    0.671624] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.671658] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

# dmesg | grep DMAR
[    0.000000] ACPI: DMAR 000000003ffe2440 00040 (v01 BOCHS  BXPCDMAR 00000001 BXPC 00000001)
[    0.671490] DMAR: No RMRR found
[    0.671493] DMAR: No ATSR found


# ll /sys/kernel/iommu_groups/
total 0
drwxr-xr-x. 3 root root 0 Aug 19 20:05 0
drwxr-xr-x. 3 root root 0 Aug 19 20:05 1
drwxr-xr-x. 3 root root 0 Aug 19 20:05 2
drwxr-xr-x. 3 root root 0 Aug 19 20:05 3
drwxr-xr-x. 3 root root 0 Aug 19 20:05 4
drwxr-xr-x. 3 root root 0 Aug 19 20:05 5
drwxr-xr-x. 3 root root 0 Aug 19 20:05 6
drwxr-xr-x. 3 root root 0 Aug 19 20:05 7

5.In the guest
# lspci | grep Eth
02:02.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)


# virsh nodedev-list --tree
...
  |             
  +- pci_0000_00_1e_0
  |   |
  |   +- pci_0000_01_00_0
  |       |
  |       +- pci_0000_02_01_0
  |       +- pci_0000_02_02_0
  |           |
  |           +- net_enp2s2_54_52_00_54_9e_f4
....


# virsh nodedev-dumpxml pci_0000_02_02_0
<device>
  <name>pci_0000_02_02_0</name>
  <path>/sys/devices/pci0000:00/0000:00:1e.0/0000:01:00.0/0000:02:02.0</path>
  <parent>pci_0000_01_00_0</parent>
  <driver>
    <name>e1000</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>2</bus>
    <slot>2</slot>
    <function>0</function>
    <product id='0x100e'>82540EM Gigabit Ethernet Controller</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <iommuGroup number='6'>
      <address domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </iommuGroup>
  </capability>
</device>

6. In the guest, I start a nested guest

# virsh start jishao
Domain jishao started

# cat vf.xml 
     <hostdev type='pci' managed='yes'>
     <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
      </source>
    </hostdev>

# virsh attach-device jishao vf.xml 
error: Failed to attach device from vf.xml
error: internal error: unable to execute QEMU command 'device_add': Device initialization failed.

Comment 16 Jingjing Shao 2016-08-25 09:06:56 UTC
One more question, Does <iommu model='intel'/>  not support migration ? Why is it a non-migratable device 'iommu-intel' ?

I can not find more info in the libvirt.org  
http://libvirt.org/formatdomain.html#elementsIommu

# virsh save r7.1  r7.1.save
error: Failed to save domain r7.1 to r7.1.save
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'iommu-intel'

Comment 17 Jingjing Shao 2016-08-25 09:08:24 UTC
Hi Ján,

Would you please help to check the question in comment15 and comment16 and update something? thank you in advance.

Comment 18 Jingjing Shao 2016-09-20 05:46:17 UTC
According to the qemu bug: https://bugzilla.redhat.com/show_bug.cgi?id=1350196
the step5 in the comment15 is enough for this bug. so I change the status to verified.

I will track the issues in the step6 and comment 16 in the rhel7.4

Comment 19 Jingjing Shao 2016-09-28 11:33:50 UTC
Some testing scenario for supplement

1. add the   <iommu model='intel'/>  to  the guest with q35 machine type
     # virsh dumpxml rhel7.3 | grep iommu
       <iommu model='intel'/>
     # virsh start q35-js
       Domain q35-js started
     # ps -ef | grep qemu | grep "iommu"
      .....-device intel-iommu .......

2. add the   <iommu model='intel'/>  to  the guest with 'pc-i440fx-rhel7.3.0' machine type
    # virsh start rhel7.3
   error: Failed to start domain rhel7.3
   error: unsupported configuration: IOMMU device: 'intel' is only supported with Q35 machines

3. coldplug reject 
    # virsh attach-device q35-js  iommu.xml  --config
error: Failed to attach device from iommu.xml
error: Operation not supported: persistent attach of device 'iommu' is not supported
    hotplug reject 
     # virsh start q35-js
     Domain q35-js started
     # virsh attach-device q35-js  iommu.xml  
error: Failed to attach device from iommu.xml
error: Operation not supported: live attach of device 'iommu' is not supported
  
4. Test with lower version of qemu-kvm-rhev
       # rpm -q qemu-kvm-rhev
       qemu-kvm-rhev-2.6.0-17.el7.x86_64
       # virsh start q35-js
error: Failed to start domain q35-js
error: internal error: qemu unexpectedly closed the monitor: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
2016-09-28T11:29:21.804359Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine

Comment 21 errata-xmlrpc 2016-11-03 18:19:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html


Note You need to log in before you can comment on or make changes to this bug.