Notes:

For enable IOMMU support of virtio devices, we need enable iommu_platform through command line: e.g -device virtio-net-pci,iommu_platform=on.

Thanks

For the cli requirements and dependencies, please see https://bugzilla.redhat.com/show_bug.cgi?id=1283104#c4

Thanks

QEMU already uses virtio-1 for devices plugged into PCIe ports so the dependency on bug 1227354 is not necessary.

Upstream patches:
https://www.redhat.com/archives/libvir-list/2017-May/msg01192.html

v2:
https://www.redhat.com/archives/libvir-list/2017-June/msg00262.html

v3:
https://www.redhat.com/archives/libvir-list/2017-June/msg00407.html

The device-iotlb, iommu_platform and ats options have been pushed upstream:

commit 27b187be3988c60cd26e08ab4bcab66bed5a3646
CommitDate: 2017-06-08 16:31:09 +0200

    conf: add iotlb attribute to iommu
    
commit 240e443afdac0df342bb462ac2754a46e6efbc23
CommitDate: 2017-06-08 16:31:28 +0200

    qemu: format device-iotlb on intel-iommu command line    

commit 15911ab8201f46b99352eea818c798828b6e4ff1
CommitDate: 2017-06-08 16:31:32 +0200

    qemuxml2xmltest: add virtio-options test
    
commit d1feb4773d41b928dc1079dfc19d17b5a0e5957b
CommitDate: 2017-06-08 16:31:48 +0200

    conf: use a leading space in virDomainVirtioNetDriverFormat
    
commit fd518643402d8233ceffe4ef28279bcce53284f6
CommitDate: 2017-06-08 16:31:54 +0200

    Add virtio-related options to interfaces

commit 82223f9364a9f47a39b7c66c241b82ae62f9fb4b
CommitDate: 2017-06-08 16:32:11 +0200

    add virtio-related options to memballoon

commit 1bc2cb3b3205dca7174147ac970e2b82c8af69da
CommitDate: 2017-06-08 16:32:27 +0200

    Add virtio-related options to disks

commit c85217cf8a81879d065b9d13e876eec141f63f6f
CommitDate: 2017-06-08 16:32:33 +0200

    Add virtio-related options to controllers

commit b10c22d9fa11e2a67eca04592688bd701700f77f
CommitDate: 2017-06-08 16:32:40 +0200

    Add virtio-related options to filesystems

commit f65db1be1200b656094180ecfdb63f8bd0158cab
CommitDate: 2017-06-08 16:32:44 +0200

    Add virtio-related options to rng devices

commit f5384fb4029a59624e728a2e0d37e6a62efbdc52
CommitDate: 2017-06-08 16:32:49 +0200

    Add virtio-related options to video

commit cc0933d3501229cdc8cf183a52a14c9b1c8de666
CommitDate: 2017-06-08 16:32:53 +0200

    Add virtio-related options to input devices

commit 56a28fbb57daa96fc7fe8f0516c8ee937ec3b39b
CommitDate: 2017-06-08 16:32:58 +0200

    qemuxml2argvtest: add virtio-options test case

commit b2cbc3a0607bf26a82911f7db6dcbc09c9bbf5e8
CommitDate: 2017-06-08 16:33:13 +0200

    qemu: format virtio-related options on the command line

git describe: v3.4.0-80-gb2cbc3a

With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, libvirt qe test 12 devices, add the summary and detailed info as below

Summary :
 1. In host, add "iommu=pt intel_iommu=on" to kernel line
 2. Add "intel_iommu=on" to kernel line of  q35  guest
 3. Add the xml as below in the guest 
   <features>
    <ioapic driver='qemu'/>
  </features>
  ...
   <iommu model='intel'>
      <driver intremap='on' iotlb='on'/>
   </iommu>


4. The test on the 9 devices as below succeed
memballoon device
rng 
virtio-serial controller 
scsi controller          
mouse                    
keyboard                 
tablet                   
passthrough              
video


i.e.
Add the iommu='on' ats='on'  on virtio-serial controller          

<controller type='virtio-serial' index='0'>
       <driver iommu='on' ats='on'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
     </controller>


# virsh start q35-js
Domain q35-js started

# ps -ef | grep qemu | grep iommu
-device virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.1,addr=0x0 

Login in guest to check the device
# lspci
01:00.0 Communication controller: Red Hat, Inc Virtio console (rev 01)


5. The test on the 2 devices as below had issues which mentioned in Comment 29 and Comment 31 

disk : start from hd， 
If add the iommu='on' ats='on' , the guest OS will report  "No bootable device" 
If delete the iommu='on' ats='on', the guest OS will start successfully.


network interface   
If add the iommu='on' ats='on' , the guest OS will hang 
If delete the iommu='on' ats='on', the guest OS will start successfully.


6. Filesystem device is not supported for libvirt side

(In reply to Jingjing Shao from comment #34)
> With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64,
> libvirt qe test 12 devices, add the summary and detailed info as below
> 
> Summary :
>  1. In host, add "iommu=pt intel_iommu=on" to kernel line
>  2. Add "intel_iommu=on" to kernel line of  q35  guest
>  3. Add the xml as below in the guest 
>    <features>
>     <ioapic driver='qemu'/>
>   </features>
>   ...
>    <iommu model='intel'>
>       <driver intremap='on' iotlb='on'/>
>    </iommu>
> 
> 
> 4. The test on the 9 devices as below succeed
> memballoon device

Have you done any operations e.g deflate/inflate? Ballon device is reported to be broken, and its iommu support will be removed.

> rng 
> virtio-serial controller 
> scsi controller          
> mouse                    
> keyboard                 
> tablet                   
> passthrough              
> video
> 
> 
> i.e.
> Add the iommu='on' ats='on'  on virtio-serial controller          
> 
> <controller type='virtio-serial' index='0'>
>        <driver iommu='on' ats='on'/>
>        <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
>      </controller>
> 
> 
> # virsh start q35-js
> Domain q35-js started
> 
> # ps -ef | grep qemu | grep iommu
> -device
> virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.1,
> addr=0x0 
> 
> Login in guest to check the device
> # lspci
> 01:00.0 Communication controller: Red Hat, Inc Virtio console (rev 01)
> 
> 
> 5. The test on the 2 devices as below had issues which mentioned in Comment
> 29 and Comment 31 
> 
> disk : start from hd， 
> If add the iommu='on' ats='on' , the guest OS will report  "No bootable
> device" 
> If delete the iommu='on' ats='on', the guest OS will start successfully.
> 

This bug has been reported upstream, please file a new bug for this.

> 
> network interface   
> If add the iommu='on' ats='on' , the guest OS will hang 

Could you show me the guest kernel version?

> If delete the iommu='on' ats='on', the guest OS will start successfully.
> 

And paste the complete qemu cli here. We've tested iommu with ats on virtio-net several time but haven't met such issue.

Thanks

> 
> 6. Filesystem device is not supported for libvirt side

(In reply to Jingjing Shao from comment #34)
> 
> 5. The test on the 2 devices as below had issues which mentioned in Comment
> 29 and Comment 31 
> 
> disk : start from hd， 
> If add the iommu='on' ats='on' , the guest OS will report  "No bootable
> device" 
> If delete the iommu='on' ats='on', the guest OS will start successfully.

Hi Jingjing,

From qemu side, virtio-blk-pci with iommu='on' ats='on' can not boot up either.

> 
> network interface   
> If add the iommu='on' ats='on' , the guest OS will hang 
> If delete the iommu='on' ats='on', the guest OS will start successfully.

The virtio-net-pci with iommu='on' ats='on' works well, see below commands: 

-device pcie-root-port,id=root.1,slot=1 \
-netdev tap,id=hostnet1,vhost=on \
-device virtio-net-pci,netdev=hostnet1,id=net1,bus=root.2,mac=88:66:da:5f:dd:12,iommu_platform=on,ats=on \

> 
> 6. Filesystem device is not supported for libvirt side

(In reply to jason wang from comment #38)

> Have you done any operations e.g deflate/inflate? Ballon device is reported
> to be broken, and its iommu support will be removed.
> 
> > rng 
> > virtio-serial controller 
> > scsi controller          
> > mouse                    
> > keyboard                 
> > tablet                   
> > passthrough              
> > video
> > 
> > 

I did not test the function of every device. I will try the other part and add the comment later.

I have tried the Ballon device , it can not work as expected. Is there already a bug to track it ？


> > 
> > 5. The test on the 2 devices as below had issues which mentioned in Comment
> > 29 and Comment 31 
> > 
> > disk : start from hd， 
> > If add the iommu='on' ats='on' , the guest OS will report  "No bootable
> > device" 
> > If delete the iommu='on' ats='on', the guest OS will start successfully.
> > 
> 
> This bug has been reported upstream, please file a new bug for this.

Thank you for your feedback and  file a new bug 
https://bugzilla.redhat.com/show_bug.cgi?id=1463163

> 
> > 
> > network interface   
> > If add the iommu='on' ats='on' , the guest OS will hang 
> 
> Could you show me the guest kernel version?

The kernel version of guest is 3.10.0-663.el7.x86_64

> 
> > If delete the iommu='on' ats='on', the guest OS will start successfully.
> > 
> 
> And paste the complete qemu cli here. We've tested iommu with ats on
> virtio-net several time but haven't met such issue.
> 
> Thanks

-device intel-iommu,intremap=on,caching-mode=on
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:98:14:7e,bus=pci.4,addr=0x0,iommu_platform=on,ats=on 

> 
> > 
> > 6. Filesystem device is not supported for libvirt side

> 
> -device intel-iommu,intremap=on,caching-mode=on
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:98:14:7e,bus=pci.4,
> addr=0x0,iommu_platform=on,ats=on 
> 
Sorry，I just find that I did not paste the whole info and lost "device-iotlb" part.

-device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on 
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d1:a7:29,bus=pci.10,addr=0x0,iommu_platform=on,ats=on

Update for other devices

(1)rng device also can not work as expected.

If add the iommu='on' ats='on' on rng devices xml part, login guest,
# cat /dev/hwrng 
-bash:cd:hwrng:no such file or directory

If delete the iommu='on' ats='on' from the guest,login guest,
the /dev/hwrng  can be read successfully


The qemu commend line
-device virtio-rng-pci,rng=objrng0,id=rng0,iommu_platform=on,ats=on,bus=pci.6,addr=0x0 



(2)scsi controller also can not work as expected.

If add the iommu='on' ats='on' on scsi controller xml part and Change the disk xml <target dev='sda' bus='sata'/>  to <target dev='sda' bus='scsi'/> , the guest can not be started successfully.

Report error: .PXE-E18: Server response timeout

If change back to <target dev='sda' bus='sata'/>, the guest can be started successfully

Thanks for the testing.

If this could be reproduced by KVM QE too, please file bug for qemu.

Thanks

Update for other devices

(3)virtio-serial controller also can not work as expected.

If add the iommu='on' ats='on' on virtio-serial controller xml part, login guest,
# ll /dev/vport1p1
ls: cannot access  vport1p1: No such file or directory

If delete the iommu='on' ats='on' from the guest,login guest,
I can find the file vport1p1


qemu command line:
-device virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.4,addr=0x0 

Other devices test successfully from libvirt side.

Per what QE(Pei) and I have tested, apparently this works for virtio-net devices, other virtio devices have not been fully verified. please see bz 1283257 for more info.

Had a quick test for virtio serial, it works with my original cli.
-netdev tap,id=hn1,script=/etc/qemu-ifup-wei,vhost=on \
-device virtio-net-pci,netdev=hn1,mac=52:54:00:11:22:10 \
-netdev tap,id=hn2,script=/etc/qemu-ifup-private1,vhost=on \
-device ioh3420,id=root.1,chassis=1 \
-device virtio-net-pci,netdev=hn2,id=v0,mq=off,mac=52:54:00:11:22:11,bus=root.1,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on \
-netdev tap,id=hn3,vhost=on,script=/etc/qemu-ifup-private2 \
-device ioh3420,id=root.2,chassis=2 \
-device virtio-net-pci,netdev=hn3,id=v1,mq=off,mac=52:54:00:11:22:12,bus=root.2,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on \
-smp 3 -m 6G -enable-kvm -cpu host -vnc 0.0.0.0:2 \
-M q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-device ioh3420,id=root.3,chassis=3 \
-device virtio-serial-pci,id=virtio-serial0,bus=root.3,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on

Pei Zhang:
Can you have a test with virtio-serial(or other devices) and paste your full cli as well, I remember we are using different cli with the same topology.

Hi Wei and Pei

I update the info again.I found different guest kernel versions have different results.

I test these concerned devices with rhel7.4 version and rhel7.3 version and summary as below which may help.

scsi controller on both versions can not work well and OS can not be accessed.


With 3.10.0-681.el7.x86_64
Network , virtio-serial controller  rng device can work well


With 3.10.0-514.el7.x86_64
virtio-serial controller, rng device can not work well as the comments as above.
For the network device, the os can access but can not get ip address

(In reply to Wei from comment #45)
> Per what QE(Pei) and I have tested, apparently this works for virtio-net
> devices, other virtio devices have not been fully verified. please see bz
> 1283257 for more info.
> 
> Had a quick test for virtio serial, it works with my original cli.
...
> 
> Pei Zhang:
> Can you have a test with virtio-serial(or other devices) and paste your full
> cli as well, I remember we are using different cli with the same topology.

Wei, below is my full command with virtio-serial, works well.

# /usr/libexec/qemu-kvm -name rhel7.4 -M q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host -m 8G \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-drive file=/mnt/nfv/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.1 \
-vnc :2 \
-monitor stdio \
-device virtio-serial-pci,id=virtio-serial0,bus=root.2,iommu_platform=on,ats=on \
-chardev socket,id=channel0,path=/tmp/helloworld0,server,nowait \
-device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm0,bus=virtio-serial0.0,id=port0

Versions:
3.10.0-685.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

For virtio-scsi-pci with iommu, qemu also hit 'No bootable device' issue, can not boot up.

Versions:
3.10.0-685.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

Full command line:
# /usr/libexec/qemu-kvm -name rhel7.4 -M q35,kernel-irqchip=split \
-device intel-iommu,device-iotlb=on,intremap \
-cpu host -m 8G \
-object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \
-numa node,memdev=mem -mem-prealloc \
-smp 4,sockets=1,cores=4,threads=1 \
-device pcie-root-port,id=root.1,slot=1 \
-device pcie-root-port,id=root.2,slot=2 \
-vnc :2 \
-monitor stdio \
-device virtio-scsi-pci,id=scsi0,bus=root.1,iommu_platform=on,ats=on \
-drive file=/mnt/nfv/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-scsi0,werror=stop,rerror=stop \
-device scsi-hd,drive=drive-virtio-scsi0,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk0 \


Jason, is this same issue with bug[1]? Or need to file a new bug to track? Thanks.
[1]Bug 1463163 - Guest OS will down when disk enable the IOMMU for Virtio

(In reply to jason wang from comment #38)
> (In reply to Jingjing Shao from comment #34)
> > With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64,
> > libvirt qe test 12 devices, add the summary and detailed info as below
> > 
> > Summary :
> >  1. In host, add "iommu=pt intel_iommu=on" to kernel line
> >  2. Add "intel_iommu=on" to kernel line of  q35  guest
> >  3. Add the xml as below in the guest 
> >    <features>
> >     <ioapic driver='qemu'/>
> >   </features>
> >   ...
> >    <iommu model='intel'>
> >       <driver intremap='on' iotlb='on'/>
> >    </iommu>
> > 
> > 
> > 4. The test on the 9 devices as below succeed
> > memballoon device
> 
> Have you done any operations e.g deflate/inflate? Ballon device is reported
> to be broken, and its iommu support will be removed.
> 

Hi Yumei,

Could you please help to test memballoon with vIOMMU? 


Thanks,
Pei

(In reply to Jingjing Shao from comment #46)
> Hi Wei and Pei
> 
> I update the info again.I found different guest kernel versions have
> different results.
> 
> I test these concerned devices with rhel7.4 version and rhel7.3 version and
> summary as below which may help.
> 
> scsi controller on both versions can not work well and OS can not be
> accessed.
> 
> 
> With 3.10.0-681.el7.x86_64
> Network , virtio-serial controller  rng device can work well
> 
> 
> With 3.10.0-514.el7.x86_64
> virtio-serial controller, rng device can not work well as the comments as
> above.
> For the network device, the os can access but can not get ip address


Update info for the memballoon device 

With 3.10.0-681.el7.x86_64(rhel7.4) , it works well and can set memory successfully

With 3.10.0-514.el7.x86_64(rhel7.3) , it fail to set memory

Thanks Wei's feedback.

Summary for this bug

With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, libvirt qe test 12 devices.

(1) Disk, scsi controller failed with rhel7.4 and rhel7.3 guest.
Have filed a qemu bug for this part for track.
https://bugzilla.redhat.com/show_bug.cgi?id=1463163


(2) memballoon device, Network device, virtio-serial controller, rng device part failed with rhel7.3 guest and succeed with rhel7.4 guest.
Have filed a kernel bug for this part for track.
https://bugzilla.redhat.com/show_bug.cgi?id=1464891

As the test as above, I change the status to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846