RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1044595 - vfio does not work "out of the box"
Summary: vfio does not work "out of the box"
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kernel
Version: 7.0
Hardware: x86_64
OS: Unspecified
medium
urgent
Target Milestone: rc
: ---
Assignee: Alex Williamson
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1044530 1044844 1046429 (view as bug list)
Depends On:
Blocks: 726818 744225 965845 1049170 1050845
TreeView+ depends on / blocked
 
Reported: 2013-12-18 15:59 UTC by Alex Williamson
Modified: 2014-11-25 13:52 UTC (History)
28 users (show)

Fixed In Version: kernel-3.10.0-73.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-06-13 10:36:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
System sosreport (5.21 MB, application/octet-stream)
2013-12-19 15:38 UTC, IBM Bug Proxy 		no flags Details
compressed libvirt.log (509.07 KB, application/gzip)
2013-12-19 15:38 UTC, IBM Bug Proxy 		no flags Details
View All

Description Alex Williamson 2013-12-18 15:59:11 UTC 
Description of problem:
libvirt tests for vfio support by looking for access("/dev/vfio/vfio", F_OK).  This file currently only exists if the vfio module is loaded.  Therefore if a user wants to make use of vfio in libvirt, they first need to pre-load the module.  It's a poor user experience if the user needs to reference documentation just to be able to use this feature.

To improve the situation, we either need to pre-load the module, which is what qemu-kvm did for things like kvm and vhost in the past or make the module auto-load, which is the new hotness.

Version-Release number of selected component (if applicable):
kernel-3.10.0-55.el7.x86_64

How reproducible:
always

Steps to Reproduce:
1. Attempt to use vfio from libvirt w/o the vfio module initially loaded
2.
3.

Actual results:


Expected results:


Additional info:
Note that vfio-pci depends on vfio so that if devices are pre-bound to vfio-pci, everything works.

To solve this, the vfio control file needs to be converted to a misc driver with static minor, which will cause udev to create the device node and auto-load vfio when it is accessed.  libvirt may need to up their test for vfio to open and test the version of vfio rather than just the existence of the device node.
Comment 2 Laine Stump 2013-12-19 15:26:50 UTC 
*** Bug 1044530 has been marked as a duplicate of this bug. ***
Comment 3 IBM Bug Proxy 2013-12-19 15:38:28 UTC 
Created attachment 839027 [details]
System sosreport

default comment by bridge
Comment 4 IBM Bug Proxy 2013-12-19 15:38:32 UTC 
Created attachment 839028 [details]
compressed libvirt.log

default comment by bridge
Comment 5 Michal Schmidt 2013-12-20 12:49:18 UTC 
*** Bug 1044844 has been marked as a duplicate of this bug. ***
Comment 6 Laine Stump 2014-01-07 14:55:55 UTC 
*** Bug 1046598 has been marked as a duplicate of this bug. ***
Comment 7 IBM Bug Proxy 2014-01-13 09:22:28 UTC 
------- Comment From onmahaja.com 2014-01-13 09:13 EDT-------
Redhat, Any updates on this bug ?
Comment 8 Andy Gospodarek 2014-01-14 21:12:30 UTC 
*** Bug 1046429 has been marked as a duplicate of this bug. ***
Comment 9 Jarod Wilson 2014-01-17 17:44:52 UTC 
Patch(es) available on kernel-3.10.0-73.el7
Comment 11 Laine Stump 2014-01-17 17:59:19 UTC 
Can you explain to someone who still doesn't understand the workflow of the upstream kernel which upstream/Fedora kernel these patches will be in?
Comment 13 Alex Williamson 2014-01-17 18:06:17 UTC 
(In reply to Laine Stump from comment #11)
> Can you explain to someone who still doesn't understand the workflow of the
> upstream kernel which upstream/Fedora kernel these patches will be in?

These are queued in linux-next for 3.14 upstream
Comment 14 Xuesong Zhang 2014-01-20 04:14:14 UTC 
hi, Alex and laine,

Tested with the following latest build, seems the vfio module will be auto-loaded while hot-plug with libvirt. 

As you know, the kvm module will be auto-loaded by kernel. Should we let the kernel auto-load the vfio related module also? 

Version:
3.10.0-73.el7.x86_64
qemu-kvm-rhev-1.5.3-38.el7.x86_64
libvirt-1.1.1-18.el7.x86_64

Steps:
1. start the os with the fixed kernel build, the vfio module didn't be auto-loaded by kernel.
# lsmod|grep vfio

2. prepare one shutooff guest with attached pci device:
# virsh dumpxml a
......
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
    </hostdev>
......

3. start the guest, but failed.
# virsh start a
error: Failed to start domain a
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to set iommu for container: Operation not permitted
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to setup container for group 2
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: vfio: failed to get group 2
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: Device initialization failed.
qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8: Device 'vfio-pci' could not be initialized

4. check the vfio module, it is auto-loaded while hot-plug the PCI device.
# lsmod|grep vfio
vfio_iommu_type1       17636  0 
vfio_pci               36474  0 
vfio                   20810  2 vfio_iommu_type1,vfio_pci

5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1 module.
# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
N
[root@xuzhang3 ~]# echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
[root@xuzhang3 ~]# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
Y

6. guest can be started successfully this time.
# virsh start a
Domain a started
Comment 15 Alex Williamson 2014-01-20 05:13:17 UTC 
(In reply to Zhang Xuesong from comment #14)
> hi, Alex and laine,
> 
> Tested with the following latest build, seems the vfio module will be
> auto-loaded while hot-plug with libvirt. 
> 
> As you know, the kvm module will be auto-loaded by kernel. Should we let the
> kernel auto-load the vfio related module also? 

I assume you're referring to vfio_iommu_type1 and vfio_pci.  The vfio module already attempts to load any available iommu backends when it loads, so vfio_iommu_type1 is already auto-loaded.  For vfio_pci, the user needs to bind devices to the driver before it's useful, so I don't see that it's all that valuable to auto-load it.  The vfio architecture also allows multiple device backends drivers, so while we only have vfio_pci today, we might someday have drivers for several devices types.  At that point we probably won't want unused devices backends auto-loaded.  libvirt already has support for auto-loading the vfio_pci module itself.
 
> Version:
> 3.10.0-73.el7.x86_64
> qemu-kvm-rhev-1.5.3-38.el7.x86_64
> libvirt-1.1.1-18.el7.x86_64
> 
> Steps:
> 1. start the os with the fixed kernel build, the vfio module didn't be
> auto-loaded by kernel.
> # lsmod|grep vfio
> 
> 2. prepare one shutooff guest with attached pci device:
> # virsh dumpxml a
> ......
>     <hostdev mode='subsystem' type='pci' managed='yes'>
>       <source>
>         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
>       </source>
>     </hostdev>
> ......
> 
> 3. start the guest, but failed.
> # virsh start a
> error: Failed to start domain a
> error: internal error: early end of file from monitor: possible problem:
> qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> vfio: failed to set iommu for container: Operation not permitted
> qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> vfio: failed to setup container for group 2
> qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> vfio: failed to get group 2
> qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> Device initialization failed.
> qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> Device 'vfio-pci' could not be initialized
> 
> 4. check the vfio module, it is auto-loaded while hot-plug the PCI device.
> # lsmod|grep vfio
> vfio_iommu_type1       17636  0 
> vfio_pci               36474  0 
> vfio                   20810  2 vfio_iommu_type1,vfio_pci
> 
> 5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1
> module.
> # cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> N
> [root@xuzhang3 ~]# echo 1 >
> /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> [root@xuzhang3 ~]# cat
> /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> Y

The expected way a customer would solve this would be with an options entry in a /etc/modprobe.d file.

> 6. guest can be started successfully this time.
> # virsh start a
> Domain a started
Comment 16 Xuesong Zhang 2014-01-20 06:28:21 UTC 
(In reply to Alex Williamson from comment #15)
> (In reply to Zhang Xuesong from comment #14)
> > hi, Alex and laine,
> > 
> > Tested with the following latest build, seems the vfio module will be
> > auto-loaded while hot-plug with libvirt. 
> > 
> > As you know, the kvm module will be auto-loaded by kernel. Should we let the
> > kernel auto-load the vfio related module also? 
> 
> I assume you're referring to vfio_iommu_type1 and vfio_pci.  The vfio module
> already attempts to load any available iommu backends when it loads, so
> vfio_iommu_type1 is already auto-loaded.  For vfio_pci, the user needs to
> bind devices to the driver before it's useful, so I don't see that it's all
> that valuable to auto-load it.  The vfio architecture also allows multiple
> device backends drivers, so while we only have vfio_pci today, we might
> someday have drivers for several devices types.  At that point we probably
> won't want unused devices backends auto-loaded.  libvirt already has support
> for auto-loading the vfio_pci module itself.
>  

Yeah, I mean the vfio, vfio_iommu_type1 and vfio_pci module.
Thanks for your details explanation.

> > Version:
> > 3.10.0-73.el7.x86_64
> > qemu-kvm-rhev-1.5.3-38.el7.x86_64
> > libvirt-1.1.1-18.el7.x86_64
> > 
> > Steps:
> > 1. start the os with the fixed kernel build, the vfio module didn't be
> > auto-loaded by kernel.
> > # lsmod|grep vfio
> > 
> > 2. prepare one shutooff guest with attached pci device:
> > # virsh dumpxml a
> > ......
> >     <hostdev mode='subsystem' type='pci' managed='yes'>
> >       <source>
> >         <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
> >       </source>
> >     </hostdev>
> > ......
> > 
> > 3. start the guest, but failed.
> > # virsh start a
> > error: Failed to start domain a
> > error: internal error: early end of file from monitor: possible problem:
> > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> > vfio: failed to set iommu for container: Operation not permitted
> > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> > vfio: failed to setup container for group 2
> > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> > vfio: failed to get group 2
> > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> > Device initialization failed.
> > qemu-kvm: -device vfio-pci,host=00:19.0,id=hostdev0,bus=pci.0,addr=0x8:
> > Device 'vfio-pci' could not be initialized
> > 
> > 4. check the vfio module, it is auto-loaded while hot-plug the PCI device.
> > # lsmod|grep vfio
> > vfio_iommu_type1       17636  0 
> > vfio_pci               36474  0 
> > vfio                   20810  2 vfio_iommu_type1,vfio_pci
> > 
> > 5. enable the "allow_unsafe_interrupts" parameter of the vfio_iommu_type1
> > module.
> > # cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> > N
> > [root@xuzhang3 ~]# echo 1 >
> > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> > [root@xuzhang3 ~]# cat
> > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
> > Y
> 
> The expected way a customer would solve this would be with an options entry
> in a /etc/modprobe.d file.

After config as your suggestion, it works more smoothly now.

1.config like the following file:
# cat /etc/modprobe.d/local.conf 
options vfio_iommu_type1 allow_unsafe_interrupts=1

2. reboot the host

3. check the vfio modules, didn't be auto-loaded while os start up.
# lsmod|grep vfio

4. start the guest with pci device, the guest can be started up successfully.
# virsh start a
Domain a started

> 
> > 6. guest can be started successfully this time.
> > # virsh start a
> > Domain a started
Comment 17 IBM Bug Proxy 2014-03-06 16:25:34 UTC 
------- Comment From whetzel.com 2014-03-06 16:12 EDT-------
Verified with 3.10.0-97.el7.x86_64
Comment 18 Ludek Smid 2014-06-13 10:36:19 UTC 
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.
Comment 21 Itzik Brown 2014-11-25 08:03:42 UTC 
Hi Alex, 
I'm testing SR-IOV with OSP6 using RHEL7 3.10.0-123.el7.x86_64.

Just to make it sure - Must I load the vfio_iommu_type1 with allow_unsafe_interrupts=1 (i.e. entry in a /etc/modprobe.d file options vfio_iommu_type1 allow_unsafe_interrupts=1) ?

Itzik
Comment 22 Alex Williamson 2014-11-25 13:52:42 UTC 
The allow_unsafe_interrupts is not automatic because enabling it potentially exposes the host to MSI attacks from the guest.  We therefore require the user to opt-in if the hardware does not support interrupt remapping.  This option should only be used on hardware lacking interrupt remapping support where the guests are relatively trusted.  If you meet these requirements, then yes, enabling it as you indicate via modprobe.d is the correct method.
Note You need to log in before you can comment on or make changes to this bug.