Bug 1283251
Summary: | [RFE] Libvirt - config Qemu to use IOMMU for Virtio | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Amnon Ilan <ailan> |
Component: | libvirt | Assignee: | Ján Tomko <jtomko> |
Status: | CLOSED ERRATA | QA Contact: | Jingjing Shao <jishao> |
Severity: | unspecified | Docs Contact: | Jiri Herrmann <jherrman> |
Priority: | high | ||
Version: | 7.3 | CC: | ailan, atragler, dyuan, jasowang, jinzhao, jishao, jsuchane, jtomko, juzhang, knoel, lmiksik, mtessun, mzhan, ovasik, pasik, peterx, pezhang, pzhang, rbalakri, wexu, xuzhang, yafu, yalzhang, yuhuang |
Target Milestone: | rc | Keywords: | FutureFeature |
Target Release: | 7.4 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | libvirt-3.2.0-10.el7 | Doc Type: | Technology Preview |
Doc Text: |
.Virtio devices can now use vIOMMU
As a Technology Preview, this update enables virtio devices to use virtual Input/Output Memory Management Unit (vIOMMU). This guarantees the security of Direct Memory Access (DMA) by allowing the device to DMA only to permitted addresses. However, note that only guest virtual machines using Red Hat Enterprise Linux 7.4 or later are able to use this feature.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2017-08-01 17:06:41 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1283250, 1350196 | ||
Bug Blocks: | 1283104, 1288337, 1395265 |
Description
Amnon Ilan
2015-11-18 14:20:09 UTC
Notes: For enable IOMMU support of virtio devices, we need enable iommu_platform through command line: e.g -device virtio-net-pci,iommu_platform=on. Thanks For the cli requirements and dependencies, please see https://bugzilla.redhat.com/show_bug.cgi?id=1283104#c4 Thanks QEMU already uses virtio-1 for devices plugged into PCIe ports so the dependency on bug 1227354 is not necessary. Upstream patches: https://www.redhat.com/archives/libvir-list/2017-May/msg01192.html The device-iotlb, iommu_platform and ats options have been pushed upstream: commit 27b187be3988c60cd26e08ab4bcab66bed5a3646 CommitDate: 2017-06-08 16:31:09 +0200 conf: add iotlb attribute to iommu commit 240e443afdac0df342bb462ac2754a46e6efbc23 CommitDate: 2017-06-08 16:31:28 +0200 qemu: format device-iotlb on intel-iommu command line commit 15911ab8201f46b99352eea818c798828b6e4ff1 CommitDate: 2017-06-08 16:31:32 +0200 qemuxml2xmltest: add virtio-options test commit d1feb4773d41b928dc1079dfc19d17b5a0e5957b CommitDate: 2017-06-08 16:31:48 +0200 conf: use a leading space in virDomainVirtioNetDriverFormat commit fd518643402d8233ceffe4ef28279bcce53284f6 CommitDate: 2017-06-08 16:31:54 +0200 Add virtio-related options to interfaces commit 82223f9364a9f47a39b7c66c241b82ae62f9fb4b CommitDate: 2017-06-08 16:32:11 +0200 add virtio-related options to memballoon commit 1bc2cb3b3205dca7174147ac970e2b82c8af69da CommitDate: 2017-06-08 16:32:27 +0200 Add virtio-related options to disks commit c85217cf8a81879d065b9d13e876eec141f63f6f CommitDate: 2017-06-08 16:32:33 +0200 Add virtio-related options to controllers commit b10c22d9fa11e2a67eca04592688bd701700f77f CommitDate: 2017-06-08 16:32:40 +0200 Add virtio-related options to filesystems commit f65db1be1200b656094180ecfdb63f8bd0158cab CommitDate: 2017-06-08 16:32:44 +0200 Add virtio-related options to rng devices commit f5384fb4029a59624e728a2e0d37e6a62efbdc52 CommitDate: 2017-06-08 16:32:49 +0200 Add virtio-related options to video commit cc0933d3501229cdc8cf183a52a14c9b1c8de666 CommitDate: 2017-06-08 16:32:53 +0200 Add virtio-related options to input devices commit 56a28fbb57daa96fc7fe8f0516c8ee937ec3b39b CommitDate: 2017-06-08 16:32:58 +0200 qemuxml2argvtest: add virtio-options test case commit b2cbc3a0607bf26a82911f7db6dcbc09c9bbf5e8 CommitDate: 2017-06-08 16:33:13 +0200 qemu: format virtio-related options on the command line git describe: v3.4.0-80-gb2cbc3a With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, libvirt qe test 12 devices, add the summary and detailed info as below Summary : 1. In host, add "iommu=pt intel_iommu=on" to kernel line 2. Add "intel_iommu=on" to kernel line of q35 guest 3. Add the xml as below in the guest <features> <ioapic driver='qemu'/> </features> ... <iommu model='intel'> <driver intremap='on' iotlb='on'/> </iommu> 4. The test on the 9 devices as below succeed memballoon device rng virtio-serial controller scsi controller mouse keyboard tablet passthrough video i.e. Add the iommu='on' ats='on' on virtio-serial controller <controller type='virtio-serial' index='0'> <driver iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </controller> # virsh start q35-js Domain q35-js started # ps -ef | grep qemu | grep iommu -device virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.1,addr=0x0 Login in guest to check the device # lspci 01:00.0 Communication controller: Red Hat, Inc Virtio console (rev 01) 5. The test on the 2 devices as below had issues which mentioned in Comment 29 and Comment 31 disk : start from hd, If add the iommu='on' ats='on' , the guest OS will report "No bootable device" If delete the iommu='on' ats='on', the guest OS will start successfully. network interface If add the iommu='on' ats='on' , the guest OS will hang If delete the iommu='on' ats='on', the guest OS will start successfully. 6. Filesystem device is not supported for libvirt side (In reply to Jingjing Shao from comment #34) > With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, > libvirt qe test 12 devices, add the summary and detailed info as below > > Summary : > 1. In host, add "iommu=pt intel_iommu=on" to kernel line > 2. Add "intel_iommu=on" to kernel line of q35 guest > 3. Add the xml as below in the guest > <features> > <ioapic driver='qemu'/> > </features> > ... > <iommu model='intel'> > <driver intremap='on' iotlb='on'/> > </iommu> > > > 4. The test on the 9 devices as below succeed > memballoon device Have you done any operations e.g deflate/inflate? Ballon device is reported to be broken, and its iommu support will be removed. > rng > virtio-serial controller > scsi controller > mouse > keyboard > tablet > passthrough > video > > > i.e. > Add the iommu='on' ats='on' on virtio-serial controller > > <controller type='virtio-serial' index='0'> > <driver iommu='on' ats='on'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x09' > function='0x0'/> > </controller> > > > # virsh start q35-js > Domain q35-js started > > # ps -ef | grep qemu | grep iommu > -device > virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.1, > addr=0x0 > > Login in guest to check the device > # lspci > 01:00.0 Communication controller: Red Hat, Inc Virtio console (rev 01) > > > 5. The test on the 2 devices as below had issues which mentioned in Comment > 29 and Comment 31 > > disk : start from hd, > If add the iommu='on' ats='on' , the guest OS will report "No bootable > device" > If delete the iommu='on' ats='on', the guest OS will start successfully. > This bug has been reported upstream, please file a new bug for this. > > network interface > If add the iommu='on' ats='on' , the guest OS will hang Could you show me the guest kernel version? > If delete the iommu='on' ats='on', the guest OS will start successfully. > And paste the complete qemu cli here. We've tested iommu with ats on virtio-net several time but haven't met such issue. Thanks > > 6. Filesystem device is not supported for libvirt side (In reply to Jingjing Shao from comment #34) > > 5. The test on the 2 devices as below had issues which mentioned in Comment > 29 and Comment 31 > > disk : start from hd, > If add the iommu='on' ats='on' , the guest OS will report "No bootable > device" > If delete the iommu='on' ats='on', the guest OS will start successfully. Hi Jingjing, From qemu side, virtio-blk-pci with iommu='on' ats='on' can not boot up either. > > network interface > If add the iommu='on' ats='on' , the guest OS will hang > If delete the iommu='on' ats='on', the guest OS will start successfully. The virtio-net-pci with iommu='on' ats='on' works well, see below commands: -device pcie-root-port,id=root.1,slot=1 \ -netdev tap,id=hostnet1,vhost=on \ -device virtio-net-pci,netdev=hostnet1,id=net1,bus=root.2,mac=88:66:da:5f:dd:12,iommu_platform=on,ats=on \ > > 6. Filesystem device is not supported for libvirt side (In reply to jason wang from comment #38) > Have you done any operations e.g deflate/inflate? Ballon device is reported > to be broken, and its iommu support will be removed. > > > rng > > virtio-serial controller > > scsi controller > > mouse > > keyboard > > tablet > > passthrough > > video > > > > I did not test the function of every device. I will try the other part and add the comment later. I have tried the Ballon device , it can not work as expected. Is there already a bug to track it ? > > > > 5. The test on the 2 devices as below had issues which mentioned in Comment > > 29 and Comment 31 > > > > disk : start from hd, > > If add the iommu='on' ats='on' , the guest OS will report "No bootable > > device" > > If delete the iommu='on' ats='on', the guest OS will start successfully. > > > > This bug has been reported upstream, please file a new bug for this. Thank you for your feedback and file a new bug https://bugzilla.redhat.com/show_bug.cgi?id=1463163 > > > > > network interface > > If add the iommu='on' ats='on' , the guest OS will hang > > Could you show me the guest kernel version? The kernel version of guest is 3.10.0-663.el7.x86_64 > > > If delete the iommu='on' ats='on', the guest OS will start successfully. > > > > And paste the complete qemu cli here. We've tested iommu with ats on > virtio-net several time but haven't met such issue. > > Thanks -device intel-iommu,intremap=on,caching-mode=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:98:14:7e,bus=pci.4,addr=0x0,iommu_platform=on,ats=on > > > > > 6. Filesystem device is not supported for libvirt side
>
> -device intel-iommu,intremap=on,caching-mode=on
> -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:98:14:7e,bus=pci.4,
> addr=0x0,iommu_platform=on,ats=on
>
Sorry,I just find that I did not paste the whole info and lost "device-iotlb" part.
-device intel-iommu,intremap=on,caching-mode=on,device-iotlb=on
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d1:a7:29,bus=pci.10,addr=0x0,iommu_platform=on,ats=on
Update for other devices (1)rng device also can not work as expected. If add the iommu='on' ats='on' on rng devices xml part, login guest, # cat /dev/hwrng -bash:cd:hwrng:no such file or directory If delete the iommu='on' ats='on' from the guest,login guest, the /dev/hwrng can be read successfully The qemu commend line -device virtio-rng-pci,rng=objrng0,id=rng0,iommu_platform=on,ats=on,bus=pci.6,addr=0x0 (2)scsi controller also can not work as expected. If add the iommu='on' ats='on' on scsi controller xml part and Change the disk xml <target dev='sda' bus='sata'/> to <target dev='sda' bus='scsi'/> , the guest can not be started successfully. Report error: .PXE-E18: Server response timeout If change back to <target dev='sda' bus='sata'/>, the guest can be started successfully Thanks for the testing. If this could be reproduced by KVM QE too, please file bug for qemu. Thanks Update for other devices (3)virtio-serial controller also can not work as expected. If add the iommu='on' ats='on' on virtio-serial controller xml part, login guest, # ll /dev/vport1p1 ls: cannot access vport1p1: No such file or directory If delete the iommu='on' ats='on' from the guest,login guest, I can find the file vport1p1 qemu command line: -device virtio-serial-pci,id=virtio-serial0,iommu_platform=on,ats=on,bus=pci.4,addr=0x0 Other devices test successfully from libvirt side. Per what QE(Pei) and I have tested, apparently this works for virtio-net devices, other virtio devices have not been fully verified. please see bz 1283257 for more info. Had a quick test for virtio serial, it works with my original cli. -netdev tap,id=hn1,script=/etc/qemu-ifup-wei,vhost=on \ -device virtio-net-pci,netdev=hn1,mac=52:54:00:11:22:10 \ -netdev tap,id=hn2,script=/etc/qemu-ifup-private1,vhost=on \ -device ioh3420,id=root.1,chassis=1 \ -device virtio-net-pci,netdev=hn2,id=v0,mq=off,mac=52:54:00:11:22:11,bus=root.1,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on \ -netdev tap,id=hn3,vhost=on,script=/etc/qemu-ifup-private2 \ -device ioh3420,id=root.2,chassis=2 \ -device virtio-net-pci,netdev=hn3,id=v1,mq=off,mac=52:54:00:11:22:12,bus=root.2,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on \ -smp 3 -m 6G -enable-kvm -cpu host -vnc 0.0.0.0:2 \ -M q35,kernel-irqchip=split \ -device intel-iommu,device-iotlb=on,intremap \ -device ioh3420,id=root.3,chassis=3 \ -device virtio-serial-pci,id=virtio-serial0,bus=root.3,disable-modern=off,disable-legacy=on,iommu_platform=on,ats=on Pei Zhang: Can you have a test with virtio-serial(or other devices) and paste your full cli as well, I remember we are using different cli with the same topology. Hi Wei and Pei I update the info again.I found different guest kernel versions have different results. I test these concerned devices with rhel7.4 version and rhel7.3 version and summary as below which may help. scsi controller on both versions can not work well and OS can not be accessed. With 3.10.0-681.el7.x86_64 Network , virtio-serial controller rng device can work well With 3.10.0-514.el7.x86_64 virtio-serial controller, rng device can not work well as the comments as above. For the network device, the os can access but can not get ip address (In reply to Wei from comment #45) > Per what QE(Pei) and I have tested, apparently this works for virtio-net > devices, other virtio devices have not been fully verified. please see bz > 1283257 for more info. > > Had a quick test for virtio serial, it works with my original cli. ... > > Pei Zhang: > Can you have a test with virtio-serial(or other devices) and paste your full > cli as well, I remember we are using different cli with the same topology. Wei, below is my full command with virtio-serial, works well. # /usr/libexec/qemu-kvm -name rhel7.4 -M q35,kernel-irqchip=split \ -device intel-iommu,device-iotlb=on,intremap \ -cpu host -m 8G \ -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \ -numa node,memdev=mem -mem-prealloc \ -smp 4,sockets=1,cores=4,threads=1 \ -device pcie-root-port,id=root.1,slot=1 \ -device pcie-root-port,id=root.2,slot=2 \ -drive file=/mnt/nfv/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \ -device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0,bus=root.1 \ -vnc :2 \ -monitor stdio \ -device virtio-serial-pci,id=virtio-serial0,bus=root.2,iommu_platform=on,ats=on \ -chardev socket,id=channel0,path=/tmp/helloworld0,server,nowait \ -device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm0,bus=virtio-serial0.0,id=port0 Versions: 3.10.0-685.el7.x86_64 qemu-kvm-rhev-2.9.0-12.el7.x86_64 For virtio-scsi-pci with iommu, qemu also hit 'No bootable device' issue, can not boot up. Versions: 3.10.0-685.el7.x86_64 qemu-kvm-rhev-2.9.0-12.el7.x86_64 Full command line: # /usr/libexec/qemu-kvm -name rhel7.4 -M q35,kernel-irqchip=split \ -device intel-iommu,device-iotlb=on,intremap \ -cpu host -m 8G \ -object memory-backend-file,id=mem,size=8G,mem-path=/dev/hugepages,share=on \ -numa node,memdev=mem -mem-prealloc \ -smp 4,sockets=1,cores=4,threads=1 \ -device pcie-root-port,id=root.1,slot=1 \ -device pcie-root-port,id=root.2,slot=2 \ -vnc :2 \ -monitor stdio \ -device virtio-scsi-pci,id=scsi0,bus=root.1,iommu_platform=on,ats=on \ -drive file=/mnt/nfv/rhel7.4_nonrt.qcow2,format=qcow2,if=none,id=drive-virtio-scsi0,werror=stop,rerror=stop \ -device scsi-hd,drive=drive-virtio-scsi0,bus=scsi0.0,scsi-id=0,lun=0,id=scsi-disk0 \ Jason, is this same issue with bug[1]? Or need to file a new bug to track? Thanks. [1]Bug 1463163 - Guest OS will down when disk enable the IOMMU for Virtio (In reply to jason wang from comment #38) > (In reply to Jingjing Shao from comment #34) > > With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, > > libvirt qe test 12 devices, add the summary and detailed info as below > > > > Summary : > > 1. In host, add "iommu=pt intel_iommu=on" to kernel line > > 2. Add "intel_iommu=on" to kernel line of q35 guest > > 3. Add the xml as below in the guest > > <features> > > <ioapic driver='qemu'/> > > </features> > > ... > > <iommu model='intel'> > > <driver intremap='on' iotlb='on'/> > > </iommu> > > > > > > 4. The test on the 9 devices as below succeed > > memballoon device > > Have you done any operations e.g deflate/inflate? Ballon device is reported > to be broken, and its iommu support will be removed. > Hi Yumei, Could you please help to test memballoon with vIOMMU? Thanks, Pei (In reply to Jingjing Shao from comment #46) > Hi Wei and Pei > > I update the info again.I found different guest kernel versions have > different results. > > I test these concerned devices with rhel7.4 version and rhel7.3 version and > summary as below which may help. > > scsi controller on both versions can not work well and OS can not be > accessed. > > > With 3.10.0-681.el7.x86_64 > Network , virtio-serial controller rng device can work well > > > With 3.10.0-514.el7.x86_64 > virtio-serial controller, rng device can not work well as the comments as > above. > For the network device, the os can access but can not get ip address Update info for the memballoon device With 3.10.0-681.el7.x86_64(rhel7.4) , it works well and can set memory successfully With 3.10.0-514.el7.x86_64(rhel7.3) , it fail to set memory Thanks Wei's feedback. Summary for this bug With libvirt-3.2.0-10.el7.x86_64 and qemu-kvm-rhev-2.9.0-10.el7.x86_64, libvirt qe test 12 devices. (1) Disk, scsi controller failed with rhel7.4 and rhel7.3 guest. Have filed a qemu bug for this part for track. https://bugzilla.redhat.com/show_bug.cgi?id=1463163 (2) memballoon device, Network device, virtio-serial controller, rng device part failed with rhel7.3 guest and succeed with rhel7.4 guest. Have filed a kernel bug for this part for track. https://bugzilla.redhat.com/show_bug.cgi?id=1464891 As the test as above, I change the status to verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 |