Bug 1044595
| Summary: | vfio does not work "out of the box" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Alex Williamson <alex.williamson> | ||||||
| Component: | kernel | Assignee: | Alex Williamson <alex.williamson> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||
| Severity: | urgent | Docs Contact: | |||||||
| Priority: | medium | ||||||||
| Version: | 7.0 | CC: | agospoda, alex.williamson, bugproxy, chayang, ddugger, dyuan, hannsj_uhl, hemanth, honli, honzhang, itbrown, jamorgan, jane.lv, jarod, john.ronciak, juzhang, jvillalo, jwilleford, knoel, laine, mzhan, qcai, ruwang, shyu, srishank, xiaolong.wang, xuzhang, xwei | ||||||
| Target Milestone: | rc | ||||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | kernel-3.10.0-73.el7 | Doc Type: | Bug Fix | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2014-06-13 10:36:19 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 726818, 744225, 965845, 1049170, 1050845 | ||||||||
| Attachments: |
|
||||||||
*** Bug 1044530 has been marked as a duplicate of this bug. *** Created attachment 839027 [details]
System sosreport
default comment by bridge
Created attachment 839028 [details]
compressed libvirt.log
default comment by bridge
*** Bug 1044844 has been marked as a duplicate of this bug. *** *** Bug 1046598 has been marked as a duplicate of this bug. *** ------- Comment From onmahaja.com 2014-01-13 09:13 EDT------- Redhat, Any updates on this bug ? *** Bug 1046429 has been marked as a duplicate of this bug. *** Patch(es) available on kernel-3.10.0-73.el7 Can you explain to someone who still doesn't understand the workflow of the upstream kernel which upstream/Fedora kernel these patches will be in? (In reply to Laine Stump from comment #11) > Can you explain to someone who still doesn't understand the workflow of the > upstream kernel which upstream/Fedora kernel these patches will be in? These are queued in linux-next for 3.14 upstream hi, Alex and laine,
Tested with the following latest build, seems the vfio module will be auto-loaded while hot-plug with libvirt.
As you know, the kvm module will be auto-loaded by kernel. Should we let the kernel auto-load the vfio related module also?
Version:
3.10.0-73.el7.x86_64
qemu-kvm-rhev-1.5.3-38.el7.x86_64
libvirt-1.1.1-18.el7.x86_64
Steps:
1. start the os with the fixed kernel build, the vfio module didn't be auto-loaded by kernel.
# lsmod|grep vfio
2. prepare one shutooff guest with attached pci device:
# virsh dumpxml a
......
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
</source>
</hostdev>
......
3. start the guest, but failed.
# virsh start a
error: Failed to start domain a
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to set iommu for container: Operation not permitted
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to setup container for group 2
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to get group 2
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: Device initialization failed.
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: Device 'vfio-pci' could not be initialized
4. check the vfio module, it is auto-loaded while hot-plug the PCI device.
# lsmod|grep vfio
vfio_iommu_type1 17636 0
vfio_pci 36474 0
vfio 20810 2 vfio_iommu_type1,vfio_pci
5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1 module.
# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
N
[root@xuzhang3 ~]# echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
[root@xuzhang3 ~]# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Y
6. guest can be started successfully this time.
# virsh start a
Domain a started
(In reply to Zhang Xuesong from comment #14) > hi, Alex and laine, > > Tested with the following latest build, seems the vfio module will be > auto-loaded while hot-plug with libvirt. > > As you know, the kvm module will be auto-loaded by kernel. Should we let the > kernel auto-load the vfio related module also? I assume you're referring to vfio_iommu_type1 and vfio_pci. The vfio module already attempts to load any available iommu backends when it loads, so vfio_iommu_type1 is already auto-loaded. For vfio_pci, the user needs to bind devices to the driver before it's useful, so I don't see that it's all that valuable to auto-load it. The vfio architecture also allows multiple device backends drivers, so while we only have vfio_pci today, we might someday have drivers for several devices types. At that point we probably won't want unused devices backends auto-loaded. libvirt already has support for auto-loading the vfio_pci module itself. > Version: > 3.10.0-73.el7.x86_64 > qemu-kvm-rhev-1.5.3-38.el7.x86_64 > libvirt-1.1.1-18.el7.x86_64 > > Steps: > 1. start the os with the fixed kernel build, the vfio module didn't be > auto-loaded by kernel. > # lsmod|grep vfio > > 2. prepare one shutooff guest with attached pci device: > # virsh dumpxml a > ...... > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > </source> > </hostdev> > ...... > > 3. start the guest, but failed. > # virsh start a > error: Failed to start domain a > error: internal error: early end of file from monitor: possible problem: > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > vfio: failed to set iommu for container: Operation not permitted > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > vfio: failed to setup container for group 2 > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > vfio: failed to get group 2 > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > Device initialization failed. > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > Device 'vfio-pci' could not be initialized > > 4. check the vfio module, it is auto-loaded while hot-plug the PCI device. > # lsmod|grep vfio > vfio_iommu_type1 17636 0 > vfio_pci 36474 0 > vfio 20810 2 vfio_iommu_type1,vfio_pci > > 5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1 > module. > # cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > N > [root@xuzhang3 ~]# echo 1 > > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > [root@xuzhang3 ~]# cat > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > Y The expected way a customer would solve this would be with an options entry in a /etc/modprobe.d file. > 6. guest can be started successfully this time. > # virsh start a > Domain a started (In reply to Alex Williamson from comment #15) > (In reply to Zhang Xuesong from comment #14) > > hi, Alex and laine, > > > > Tested with the following latest build, seems the vfio module will be > > auto-loaded while hot-plug with libvirt. > > > > As you know, the kvm module will be auto-loaded by kernel. Should we let the > > kernel auto-load the vfio related module also? > > I assume you're referring to vfio_iommu_type1 and vfio_pci. The vfio module > already attempts to load any available iommu backends when it loads, so > vfio_iommu_type1 is already auto-loaded. For vfio_pci, the user needs to > bind devices to the driver before it's useful, so I don't see that it's all > that valuable to auto-load it. The vfio architecture also allows multiple > device backends drivers, so while we only have vfio_pci today, we might > someday have drivers for several devices types. At that point we probably > won't want unused devices backends auto-loaded. libvirt already has support > for auto-loading the vfio_pci module itself. > Yeah, I mean the vfio, vfio_iommu_type1 and vfio_pci module. Thanks for your details explanation. > > Version: > > 3.10.0-73.el7.x86_64 > > qemu-kvm-rhev-1.5.3-38.el7.x86_64 > > libvirt-1.1.1-18.el7.x86_64 > > > > Steps: > > 1. start the os with the fixed kernel build, the vfio module didn't be > > auto-loaded by kernel. > > # lsmod|grep vfio > > > > 2. prepare one shutooff guest with attached pci device: > > # virsh dumpxml a > > ...... > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > > </source> > > </hostdev> > > ...... > > > > 3. start the guest, but failed. > > # virsh start a > > error: Failed to start domain a > > error: internal error: early end of file from monitor: possible problem: > > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > > vfio: failed to set iommu for container: Operation not permitted > > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > > vfio: failed to setup container for group 2 > > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > > vfio: failed to get group 2 > > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > > Device initialization failed. > > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: > > Device 'vfio-pci' could not be initialized > > > > 4. check the vfio module, it is auto-loaded while hot-plug the PCI device. > > # lsmod|grep vfio > > vfio_iommu_type1 17636 0 > > vfio_pci 36474 0 > > vfio 20810 2 vfio_iommu_type1,vfio_pci > > > > 5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1 > > module. > > # cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > > N > > [root@xuzhang3 ~]# echo 1 > > > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > > [root@xuzhang3 ~]# cat > > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts > > Y > > The expected way a customer would solve this would be with an options entry > in a /etc/modprobe.d file. After config as your suggestion, it works more smoothly now. 1.config like the following file: # cat /etc/modprobe.d/local.conf options vfio_iommu_type1 allow_unsafe_interrupts=1 2. reboot the host 3. check the vfio modules, didn't be auto-loaded while os start up. # lsmod|grep vfio 4. start the guest with pci device, the guest can be started up successfully. # virsh start a Domain a started > > > 6. guest can be started successfully this time. > > # virsh start a > > Domain a started ------- Comment From whetzel.com 2014-03-06 16:12 EDT------- Verified with 3.10.0-97.el7.x86_64 This request was resolved in Red Hat Enterprise Linux 7.0. Contact your manager or support representative in case you have further questions about the request. Hi Alex, I'm testing SR-IOV with OSP6 using RHEL7 3.10.0-123.el7.x86_64. Just to make it sure - Must I load the vfio_iommu_type1 with allow_unsafe_interrupts=1 (i.e. entry in a /etc/modprobe.d file options vfio_iommu_type1 allow_unsafe_interrupts=1) ? Itzik The allow_unsafe_interrupts is not automatic because enabling it potentially exposes the host to MSI attacks from the guest. We therefore require the user to opt-in if the hardware does not support interrupt remapping. This option should only be used on hardware lacking interrupt remapping support where the guests are relatively trusted. If you meet these requirements, then yes, enabling it as you indicate via modprobe.d is the correct method. |
Description of problem: libvirt tests for vfio support by looking for access("/dev/vfio/vfio", F_OK). This file currently only exists if the vfio module is loaded. Therefore if a user wants to make use of vfio in libvirt, they first need to pre-load the module. It's a poor user experience if the user needs to reference documentation just to be able to use this feature. To improve the situation, we either need to pre-load the module, which is what qemu-kvm did for things like kvm and vhost in the past or make the module auto-load, which is the new hotness. Version-Release number of selected component (if applicable): kernel-3.10.0-55.el7.x86_64 How reproducible: always Steps to Reproduce: 1. Attempt to use vfio from libvirt w/o the vfio module initially loaded 2. 3. Actual results: Expected results: Additional info: Note that vfio-pci depends on vfio so that if devices are pre-bound to vfio-pci, everything works. To solve this, the vfio control file needs to be converted to a misc driver with static minor, which will cause udev to create the device node and auto-load vfio when it is accessed. libvirt may need to up their test for vfio to open and test the version of vfio rather than just the existence of the device node.