Bug 1076959 (libvirt-api-hw_io_virt)
Summary: | Expose host hardware support for I/O virtualization via libvirt API | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Stephen Gordon <sgordon> | ||||
Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> | ||||
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | high | ||||||
Version: | 7.0 | CC: | dyuan, honzhang, jiahu, jmiao, mprivozn, mzhan, rbalakri, sgordon, tvvcox, weizhan | ||||
Target Milestone: | rc | Keywords: | FutureFeature, Upstream | ||||
Target Release: | --- | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-1.2.7-1.el7 | Doc Type: | Enhancement | ||||
Doc Text: |
Feature:
Expose host virtualization capabilities
Reason:
With the virtualization boom, most hardware vendors (notably CPU vendors) started to build products which more or less enhance virtualization (in most cases it's about speeding up virtualizaiton by offloading some operations onto HW). But not all hosts have these features, so if management application is deciding on which host a guest should run, it needs to know which hosts have the HW support, and which don't.
Result:
Libvirt's capabilities XML was adjusted to expose host HW virtualization support.
|
Story Points: | --- | ||||
Clone Of: | |||||||
: | 1136148 (view as bug list) | Environment: | |||||
Last Closed: | 2015-03-05 07:31:43 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1078542, 1093840, 1113520, 1136148 | ||||||
Attachments: |
|
Description
Stephen Gordon
2014-03-16 18:56:59 UTC
(In reply to Stephen Gordon from comment #0) > Description of problem: > > Examine whether it's possible to *reliably* determine host support for I/O > virtualization and reflect the host's support or lack thereof for this > technology via the API. > > Additional Information: > > It is not possible to know through libvirt if HW support for I/O > virtualization (second-level memory address translation services for I/O) is > enabled. > Command for solution in Linux: > $ cat /var/log/dmesg|grep -e IOMMU -e AMD-Vi > If the result is not null, then it is supported. Sigh, this is not a nice way. Is there something better? E.g. a file that we can parse to get the same answer? That's what we want to find out ;). Dan seemed to think we may be able to determine it now using VFIO? "Historically kernel provided no practical way to determine this. The suggested "solution" of grepping dmesg is not a satisfactory solution. Perhaps with VFIO we can infact now get a good indicator from the kernel if this is properly supported" (In reply to Stephen Gordon from comment #3) > That's what we want to find out ;). Dan seemed to think we may be able to > determine it now using VFIO? Well, I should probably be more verbose why grepping dmesg is not a good idea. But at first let me state that libvirt is perfectly fine with running a command and then parsing its output. We're doing it already. However, having said that, dmesg is a special case. The kernel log bugger can be cleared out (dmesg -C) or while it is a ring buffer the info that we are looking for might get overwritten with newer messages. So in the result, libvirt would return misleading values. This situation doesn't apply to other utilities like hugeadm though. So maybe the best solution would be to drag in kernel guys and ask them. Nobody is suggesting that grepping dmesg is the desired solution, it was simply provided by the customer as what they do today for determining whether IOMMU is available - which they want to be able to get via Libvirt instead. I've just proposed patches upstream: https://www.redhat.com/archives/libvir-list/2014-May/msg00991.html This is exposed in the libvirt's capabilities xml: # virsh capabilities <capabilities> <host> <uuid>c612af80-f8bd-11df-a08b-9a2ed278ba85</uuid> ... <kvm>true</kvm> <vfio>true</vfio> </host> ... </capabilities> Another attempt: https://www.redhat.com/archives/libvir-list/2014-June/msg00984.html And another one: https://www.redhat.com/archives/libvir-list/2014-June/msg01494.html So I've juts pushed patches upstream: commit 94e3f23e8a72dadb2ab60d8a00d77fb9c49de87e Author: Michal Privoznik <mprivozn> AuthorDate: Wed Jun 25 18:39:29 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:38 2014 +0200 qemu: Implement virConnectGetDomainCapabilities So far only information on disks and host devices are exposed in the capabilities XML. Well, at least something. Even a new test is introduced. The qemu capabilities are stolen from already existing qemucapabilities test. There's one tricky point though. Functions that checks host's KVM and VFIO capabilities, are impossible to mock currently. So in the test, we are setting the capabilities by hand. Signed-off-by: Michal Privoznik <mprivozn> commit cb01d2b5b11a21082e031b0c3000124ed47c88b7 Author: Michal Privoznik <mprivozn> AuthorDate: Mon Jun 30 16:39:18 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:38 2014 +0200 qemu_capabilities: Introduce virQEMUCapsGetDefaultMachine Sometimes it may be useful to get a default machine for given qemu binary. Fortunately, the default machine is stored always on the first position in the supported machines array. Signed-off-by: Michal Privoznik <mprivozn> commit 69f92a87c262ffdfcd1e9485c8eabef67d74aade Author: Michal Privoznik <mprivozn> AuthorDate: Mon Jun 30 16:31:51 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:38 2014 +0200 qemu_capabilities: Introduce virQEMUCapsIsMachineSupported This internal API is meant to answer the question 'Is this machine type supported by given qemu?'. Signed-off-by: Michal Privoznik <mprivozn> commit 4e30af38536e5f11cac66170a8e5e3af022b26b3 Author: Michal Privoznik <mprivozn> AuthorDate: Mon Jun 30 16:12:07 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:37 2014 +0200 qemu_capabilities: Introduce virQEMUCapsCacheLookupByArch The API may come handy if somebody has an architecture and wants to look through available qemus if the architecture is supported or not. Signed-off-by: Michal Privoznik <mprivozn> commit 3ecb9e7631f9f6a82e91e13abe6f0f700ccc4238 Author: Michal Privoznik <mprivozn> AuthorDate: Fri Jun 27 16:39:27 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:37 2014 +0200 tests: Move qemu caps XML parsing into shared unit Later on, we the qemu capabilities XML parsing code may come handy so instead of duplicating the code make the already existing one shared. By the same time, make the function accept file name instead of XML document stored already in memory. Signed-off-by: Michal Privoznik <mprivozn> commit e9f2929f41e353e9e11c0ddb86f66c845066511c Author: Michal Privoznik <mprivozn> AuthorDate: Wed Jun 25 17:56:20 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:37 2014 +0200 virsh: expose virConnectGetDomainCapabilities The API is exposed under 'domcapabilities' command. Currently, with the variety of drivers that libvirt supports, none of the command arguments is obligatory, but all are optional instead. Signed-off-by: Michal Privoznik <mprivozn> commit f409df4de1bcde7bd0732b15f78a2db918dad03b Author: Michal Privoznik <mprivozn> AuthorDate: Wed Jun 25 17:05:20 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:37 2014 +0200 Introduce virConnectGetDomainCapabilities The API should expose the information contained in virDomainCapsPtr. Signed-off-by: Michal Privoznik <mprivozn> commit 614581f32ba14311b06821e6f8279d431a6ca840 Author: Michal Privoznik <mprivozn> AuthorDate: Wed Jun 25 13:24:53 2014 +0200 Commit: Michal Privoznik <mprivozn> CommitDate: Thu Jul 3 12:22:37 2014 +0200 Introduce domain_capabilities This new module holds and formats capabilities for emulator. If you are about to create a new domain, you may want to know what is the host or hypervisor capable of. To make sure we don't regress on the XML, the formatting is not something left for each driver to implement, rather there's general format function. The domain capabilities is a lockable object (even though the locking is not necessary yet) which uses reference counter. Signed-off-by: Michal Privoznik <mprivozn> v1.2.6-30-g94e3f23 I'm trying to verify the bug using below steps. <1> Enabled IOMMU [root@ibm-x3850x5-06 ~]# rpm -q libvirt qemu-kvm libvirt-1.2.8-5.el7.x86_64 qemu-kvm-1.5.3-75.el7.x86_64 [root@ibm-x3850x5-06 ~]# dmesg | grep -i intel-iommu [ 0.000000] Intel-IOMMU: enabled [root@ibm-x3850x5-06 ~]# virsh domcapabilities <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>qemu</domain> <machine>pc-i440fx-rhel7.0.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'> <value>default</value> <value>vfio</value> </enum> </hostdev> </devices> </domainCapabilities> <2> Disabled IOMMU [root@ibm-x3850x5-06 ~]# dmesg | grep -i intel-iommu [ 0.000000] Intel-IOMMU: disabled [root@ibm-x3850x5-06 ~]# virsh domcapabilities <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>qemu</domain> <machine>pc-i440fx-rhel7.0.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'/> </hostdev> </devices> </domainCapabilities> <3> With sub-options during IOMMU enabled [root@ibm-x3850x5-06 ~]# virsh domcapabilities --emulatorbin /usr/libexec/qemu-kvm --virttype kvm --arch x86_64 --machine pc-i440fx-rhel7.0.0 <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>kvm</domain> <machine>pc-i440fx-rhel7.0.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <value>/usr/share/OVMF/OVMF_CODE.fd</value> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'> <value>default</value> <value>vfio</value> </enum> </hostdev> </devices> </domainCapabilities> Is it enough for verifying the bug? (In reply to Hu Jianwei from comment #13) > I'm trying to verify the bug using below steps. > > <1> Enabled IOMMU > [root@ibm-x3850x5-06 ~]# rpm -q libvirt qemu-kvm > libvirt-1.2.8-5.el7.x86_64 > qemu-kvm-1.5.3-75.el7.x86_64 > > [root@ibm-x3850x5-06 ~]# dmesg | grep -i intel-iommu > [ 0.000000] Intel-IOMMU: enabled Okay, so this suggest KVM should be accessible ... > [root@ibm-x3850x5-06 ~]# virsh domcapabilities > <domainCapabilities> > <enum name='pciBackend'> > <value>default</value> > <value>vfio</value> > </enum> .. but KVM is not enumerated here. I've seen such report somewhere, but the reporter went unresponsive. Is there the /dev/kvm file and is it accessible? > </hostdev> > </devices> > </domainCapabilities> > > <2> Disabled IOMMU > [root@ibm-x3850x5-06 ~]# dmesg | grep -i intel-iommu > [ 0.000000] Intel-IOMMU: disabled > [root@ibm-x3850x5-06 ~]# virsh domcapabilities > <domainCapabilities> > <path>/usr/libexec/qemu-kvm</path> > <enum name='pciBackend'/> This is correct. Without IOMMU nor KVM nor VFIO is possible. > </hostdev> > </devices> > </domainCapabilities> > > <3> With sub-options during IOMMU enabled > [root@ibm-x3850x5-06 ~]# virsh domcapabilities --emulatorbin > /usr/libexec/qemu-kvm --virttype kvm --arch x86_64 --machine > pc-i440fx-rhel7.0.0 > <domainCapabilities> > <enum name='pciBackend'> > <value>default</value> > <value>vfio</value> > </enum> Again, this is the same case as in 1). Moreover, I've written a small test program to see what's going on. Can you please compile it and run on the host that is not reporting KVM? Thank you. Created attachment 947515 [details]
test2.c
A small test program to check KVM extension availability.
(In reply to Michal Privoznik from comment #15) > Created attachment 947515 [details] > test2.c > > A small test program to check KVM extension availability. [root@ibm-x3850x5-06 ~]# lsmod | grep kvm kvm_intel 147482 4 kvm 455071 1 kvm_intel [root@ibm-x3850x5-06 ~]# ll /dev/kvm crw-rw-rw-. 1 root kvm 10, 232 Oct 16 15:34 /dev/kvm [root@ibm-x3850x5-06 ~]# ./small Trying to open /dev/kvm ... success kvmfd=3 Querying KVM_CAP_IOMMU ... failed, ioctl ret=0 (In reply to Hu Jianwei from comment #16) > (In reply to Michal Privoznik from comment #15) > > Created attachment 947515 [details] > > test2.c > > > > A small test program to check KVM extension availability. > > [root@ibm-x3850x5-06 ~]# lsmod | grep kvm > kvm_intel 147482 4 > kvm 455071 1 kvm_intel > > [root@ibm-x3850x5-06 ~]# ll /dev/kvm > crw-rw-rw-. 1 root kvm 10, 232 Oct 16 15:34 /dev/kvm > > [root@ibm-x3850x5-06 ~]# ./small > Trying to open /dev/kvm ... success kvmfd=3 > Querying KVM_CAP_IOMMU ... failed, ioctl ret=0 Correct. So it proves that kernel misses the KVM_CAP_IOMMU and hence libvirt won't report it. Awesome! So I think that makes this bug VERIFIED. Although to be 100% sure I'd find a machine that does report KVM and do the testing with disabling/enabling IOMMU. Verifying the bug using qemu-kvm-rhev build. [root@localhost ~]# rpm -q libvirt qemu-kvm-rhev libvirt-1.2.8-5.el7.x86_64 qemu-kvm-rhev-2.1.2-3.el7.x86_64 1. Disabled IOMMU [root@localhost ~]# dmesg | grep -i intel-iommu [root@localhost ~]# virsh domcapabilities <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>qemu</domain> <machine>pc-i440fx-rhel7.1.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'/> </hostdev> </devices> </domainCapabilities> 2. Enabled IOMMU [root@localhost ~]# dmesg | grep -i intel-iommu [ 0.000000] Intel-IOMMU: enabled [root@localhost ~]# virsh domcapabilities <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>qemu</domain> <machine>pc-i440fx-rhel7.1.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'> <value>default</value> <value>vfio</value> </enum> </hostdev> </devices> </domainCapabilities> 3. With sub-options during IOMMU enabled [root@localhost ~]# virsh domcapabilities --emulatorbin /usr/libexec/qemu-kvm --virttype kvm --arch x86_64 --machine pc-i440fx-rhel7.0.0 <domainCapabilities> <path>/usr/libexec/qemu-kvm</path> <domain>kvm</domain> <machine>pc-i440fx-rhel7.0.0</machine> <arch>x86_64</arch> <vcpu max='240'/> <os supported='yes'> <loader supported='yes'> <enum name='type'> <value>rom</value> <value>pflash</value> </enum> <enum name='readonly'> <value>yes</value> <value>no</value> </enum> </loader> </os> <devices> <disk supported='yes'> <enum name='diskDevice'> <value>disk</value> <value>cdrom</value> <value>floppy</value> <value>lun</value> </enum> <enum name='bus'> <value>ide</value> <value>fdc</value> <value>scsi</value> <value>virtio</value> <value>usb</value> </enum> </disk> <hostdev supported='yes'> <enum name='mode'> <value>subsystem</value> </enum> <enum name='startupPolicy'> <value>default</value> <value>mandatory</value> <value>requisite</value> <value>optional</value> </enum> <enum name='subsysType'> <value>usb</value> <value>pci</value> <value>scsi</value> </enum> <enum name='capsType'/> <enum name='pciBackend'> <value>default</value> <value>vfio</value> </enum> </hostdev> </devices> </domainCapabilities> 4. Using Michal's C tool to confirm the "KVM" backend driver [root@localhost ~]# ./small Trying to open /dev/kvm ... success kvmfd=3 Querying KVM_CAP_IOMMU ... failed, ioctl ret=0 According to comment 18, this bug is VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0323.html |