Bug 1235581

Is this command line for q35 machines all that is needed here?
-device intel-iommu

It seems broken upstream unless I apply the following fix:
https://lists.gnu.org/archive/html/qemu-devel/2016-06/msg03548.html

qemu-git: Option '-device intel-iommu' cannot be handled by this machine

Proposed upstream patches:
https://www.redhat.com/archives/libvir-list/2016-June/msg01645.html

v2: https://www.redhat.com/archives/libvir-list/2016-July/msg00334.html

Pushed upstream as:
commit ea0ed35d6efcba9e79c76d7b57959f553b76224a
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-12 12:36:13 +0200

    Introduce <iommu> device
    
    A device with an attribute 'model', with just one model
    so far:
    
    <devices>
      ...
      <iommu model='intel'/>
    </devices>
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

commit 8e7e79738d47d7881eb6ccc61ddcb0cfba418d94
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-12 12:36:13 +0200

    Add QEMU_CAPS_DEVICE_INTEL_IOMMU
    
    Check whether QEMU supports -device intel-iommu
    
    Note that the presence of this option does not mean that it's
    usable because of a bug in earlier QEMU versions, but it's
    better than nothing.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

commit 4c382376da95bf84c9903e62c714ffa33f85d404
Author:     Ján Tomko <jtomko>
CommitDate: 2016-07-12 12:36:13 +0200

    qemu: format intel-iommu on the command line
    
    <devices>
      <iommu model='intel'/>
    </devices>
    
    results in:
    
    -device intel-iommu
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1235580

git describe: v2.0.0-126-g4c38237

Hi Ján,

I rpmbuild the newest patches, try to test this issue and get error as Comment 4.  Is it caused by the qemu-kvm bug https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? 

# rpm -qa | grep qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-11.el7.x86_64
# rpm -q libvirt
libvirt-2.1.0-1.el7.x86_64



# virsh dumpxml r7.1
....
<os>
    <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type>
    <boot dev='hd'/>
</os>
....
<controller type='pci' index='0' model='pcie-root'/>
....
<iommu model='intel'/>
...


# virsh start r7.1
error: Failed to start domain r7.1
error: internal error: process exited while connecting to monitor: 2016-07-18T11:12:20.827417Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine

Hi Ján,

Would you please help to look at the error and question in the comment10 ？ Thank you in advanced

(In reply to Jingjing Shao from comment #10)
> I rpmbuild the newest patches, try to test this issue and get error as
> Comment 4.  Is it caused by the qemu-kvm bug
> https://bugzilla.redhat.com/show_bug.cgi?id=1235580 ? 
> 

Bug 1350196 is tracking the switch from -machine,iommu=on to -device.

Hi Ján,
I try to verify this bug，but I get result as below. I have three question about the result.Can you help to  check it and update something? thank you in advance.

I find that in the guest the NIC device share the same IOMMU group as the step 5. Is is right?
If so , the device can not be assigned to the nested guest.

Or will it be supported in https://bugzilla.redhat.com/show_bug.cgi?id=1283251 ?

# rpm -q libvirt
libvirt-2.0.0-5.el7.x86_64
# rpm -q qemu-kvm-rhev
qemu-kvm-rhev-2.6.0-20.el7.x86_64

1.prepare a guest with q35 machine type as below and add <iommu model='intel'/>

# virsh dumpxml q35-js
<domain type='kvm' id='12'>
  <name>q35-js</name>
  <uuid>251e7c62-3df4-4354-9a5b-0c7972035393</uuid>
.....
<os>
    <type arch='x86_64' machine='pc-q35-rhel7.3.0'>hvm</type>  -
    <boot dev='hd'/>
</os>
.....
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <feature policy='require' name='vmx'/>
  </cpu>
...
<devices>
 ...
<controller type='pci' index='0' model='pcie-root'/>
<interface type='network'>
      <mac address='54:52:00:54:9e:f4'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
</interface>

<iommu model='intel'/>
</devices>


2. start the guest
Already add the "intel_iommu=on" in kernel commend line
# virsh start q35-js
Domain q35-js started

3.Check the qemu commend line 
# ps -ef | grep qemu | grep iommu
....-device intel-iommu.....

4.In the guest
# dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.110758] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap 12008c22260206 ecap f02
[    0.671621] IOMMU 0 0xfed90000: using Queued invalidation
[    0.671623] IOMMU: Setting RMRR:
[    0.671624] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.671658] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

# dmesg | grep DMAR
[    0.000000] ACPI: DMAR 000000003ffe2440 00040 (v01 BOCHS  BXPCDMAR 00000001 BXPC 00000001)
[    0.671490] DMAR: No RMRR found
[    0.671493] DMAR: No ATSR found


# ll /sys/kernel/iommu_groups/
total 0
drwxr-xr-x. 3 root root 0 Aug 19 20:05 0
drwxr-xr-x. 3 root root 0 Aug 19 20:05 1
drwxr-xr-x. 3 root root 0 Aug 19 20:05 2
drwxr-xr-x. 3 root root 0 Aug 19 20:05 3
drwxr-xr-x. 3 root root 0 Aug 19 20:05 4
drwxr-xr-x. 3 root root 0 Aug 19 20:05 5
drwxr-xr-x. 3 root root 0 Aug 19 20:05 6
drwxr-xr-x. 3 root root 0 Aug 19 20:05 7

5.In the guest
# lspci | grep Eth
02:02.0 Ethernet controller: Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03)


# virsh nodedev-list --tree
...
  |             
  +- pci_0000_00_1e_0
  |   |
  |   +- pci_0000_01_00_0
  |       |
  |       +- pci_0000_02_01_0
  |       +- pci_0000_02_02_0
  |           |
  |           +- net_enp2s2_54_52_00_54_9e_f4
....


# virsh nodedev-dumpxml pci_0000_02_02_0
<device>
  <name>pci_0000_02_02_0</name>
  <path>/sys/devices/pci0000:00/0000:00:1e.0/0000:01:00.0/0000:02:02.0</path>
  <parent>pci_0000_01_00_0</parent>
  <driver>
    <name>e1000</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>2</bus>
    <slot>2</slot>
    <function>0</function>
    <product id='0x100e'>82540EM Gigabit Ethernet Controller</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
    <iommuGroup number='6'>
      <address domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
      <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
      <address domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </iommuGroup>
  </capability>
</device>

6. In the guest, I start a nested guest

# virsh start jishao
Domain jishao started

# cat vf.xml 
     <hostdev type='pci' managed='yes'>
     <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
      </source>
    </hostdev>

# virsh attach-device jishao vf.xml 
error: Failed to attach device from vf.xml
error: internal error: unable to execute QEMU command 'device_add': Device initialization failed.

One more question， Does <iommu model='intel'/>  not support migration ? Why is it a non-migratable device 'iommu-intel' ?

I can not find more info in the libvirt.org  
http://libvirt.org/formatdomain.html#elementsIommu

# virsh save r7.1  r7.1.save
error: Failed to save domain r7.1 to r7.1.save
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'iommu-intel'

Hi Ján,

Would you please help to check the question in comment15 and comment16 and update something? thank you in advance.

According to the qemu bug: https://bugzilla.redhat.com/show_bug.cgi?id=1350196
the step5 in the comment15 is enough for this bug. so I change the status to verified.

I will track the issues in the step6 and comment 16 in the rhel7.4

Some testing scenario for supplement

1. add the   <iommu model='intel'/>  to  the guest with q35 machine type
     # virsh dumpxml rhel7.3 | grep iommu
       <iommu model='intel'/>
     # virsh start q35-js
       Domain q35-js started
     # ps -ef | grep qemu | grep "iommu"
      .....-device intel-iommu .......

2. add the   <iommu model='intel'/>  to  the guest with 'pc-i440fx-rhel7.3.0' machine type
    # virsh start rhel7.3
   error: Failed to start domain rhel7.3
   error: unsupported configuration: IOMMU device: 'intel' is only supported with Q35 machines

3. coldplug reject 
    # virsh attach-device q35-js  iommu.xml  --config
error: Failed to attach device from iommu.xml
error: Operation not supported: persistent attach of device 'iommu' is not supported
    hotplug reject 
     # virsh start q35-js
     Domain q35-js started
     # virsh attach-device q35-js  iommu.xml  
error: Failed to attach device from iommu.xml
error: Operation not supported: live attach of device 'iommu' is not supported
  
4. Test with lower version of qemu-kvm-rhev
       # rpm -q qemu-kvm-rhev
       qemu-kvm-rhev-2.6.0-17.el7.x86_64
       # virsh start q35-js
error: Failed to start domain q35-js
error: internal error: qemu unexpectedly closed the monitor: warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
warning: host doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
2016-09-28T11:29:21.804359Z qemu-kvm: Option '-device intel-iommu' cannot be handled by this machine

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html