Bug 1001738

This bug is now partially fixed upstream by:

commit ef29de14c37d14abc546e90555a0093797facfdd
Author: Peter Krempa <pkrempa>
Date:   Wed Sep 18 16:23:14 2013 +0200

    qemu: Wire up better early error reporting
    
    The previous patches added infrastructure to report better errors from
    monitor in some cases. This patch finalizes this "feature" by enabling
    this enhanced error reporting on early phases of VM startup. In these
    phases the possibility of qemu producing a useful error message is
    really high compared to running it during the whole life cycle. After
    the start up is complete, the feature is disabled to provide the usual
    error messages so that users are not confused by possibly irrelevant
    messages that may be in the domain log.
    
    The original motivation to do this enhancement is to capture errors when
    using VFIO device passthrough, where qemu reports errors after the
    monitor is initialized and the existing error catching code couldn't
    catch this producing a unhelpful message:
    
     # virsh start test
     error: Failed to start domain test
     error: Unable to read from monitor: Connection reset by peer
    
    With this change, the message is changed to:
    
     # virsh start test
     error: Failed to start domain test
     error: internal error: early end of file from monitor: possible problem:
     qemu-system-x86_64: -device vfio-pci,host=00:1a.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: error, group 8 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
     qemu-system-x86_64: -device vfio-pci,host=00:1a.0,id=hostdev0,bus=pci.0,addr=0x5: vfio: failed to get group 8
     qemu-system-x86_64: -device vfio-pci,host=00:1a.0,id=hostdev0,bus=pci.0,addr=0x5: Device 'vfio-pci' could not be initialized

commit 90139a6236eb20a5cd2595af39ea11adae7d54de
Author: Peter Krempa <pkrempa>
Date:   Wed Sep 18 16:17:39 2013 +0200

    qemu: monitor: Produce better errors on monitor hangup
    
    Change the monitor error code to add the ability to access the qemu log
    file using a file descriptor so that we can dig in it for a more useful
    error message. The error is now logged on monitor hangups and overwrites
    a possible lesser error. A hangup on the monitor usualy means that qemu
    has crashed and there's a significant chance it produced a useful error
    message.
    
    The functionality will be latent until the next patch.

commit 8519e9ecdcc26f9753dbd85e897daabbaa82dee8
Author: Peter Krempa <pkrempa>
Date:   Wed Sep 18 16:12:17 2013 +0200

    qemu: monitor: Add infrastructure to access VM logs for better err msgs
    
    Early VM startup errors usually produce a better error message in the
    machine log file. Currently we were accessing it only when the process
    exited during certain phases of startup. This will help adding a more
    comprehensive error extraction for early qemu startup phases.
    
    This patch adds infrastructure to keep a file descriptor for the machine
    log file that will be used in case an error happens.

commit 310651a5e346b23db9015061452b1887335aed67
Author: Peter Krempa <pkrempa>
Date:   Wed Sep 18 14:43:52 2013 +0200

    qemu_process: Make qemuProcessReadLog() more versatile and reusable
    
    Teach the function to skip character device definitions printed by qemu
    at startup in addition to libvirt log messages and make it usable from
    outside of qemu_process.c. Also add documentation about the func.

*** Bug 979988 has been marked as a duplicate of this bug. ***

Another commit required:

commit 59e21e973fdbfc9065ff5aa421ae36dbd1c4073a
Author: Peter Krempa <pkrempa>
Date:   Mon Sep 30 11:47:47 2013 +0200

    qemu: process: Silence coverity warning when rewinding log file
    
    The change in ef29de14c37d14abc546e90555a0093797facfdd that introduced
    better error logging from qemu introduced a warning from coverity about
    unused return value from lseek. Silence this warning and fix typo in the
    corresponding error message.
    
    Reported by: John Ferlan

Rest of the patches:

commit f094aaac48a6bbeef87a4df0345dfe2964297298
Author: Peter Krempa <pkrempa>
Date:   Fri Sep 20 10:39:51 2013 +0200

    qemu: Prefer VFIO for PCI device passthrough
    
    Prefer using VFIO (if available) to the legacy KVM device passthrough.
    
    With this patch a PCI passthrough device without the driver configured
    will be started with VFIO if it's available on the host. If not legacy
    KVM passthrough is checked and error is reported if it's not available.

commit 467b561ac2ca7cb968d7a1d781e715cdd7bf3d14
Author: Peter Krempa <pkrempa>
Date:   Thu Sep 19 17:01:17 2013 +0200

    qemu: hostdev: Add checks if PCI passthrough is available in the host
    
    Add code to check availability of PCI passhthrough using VFIO and the
    legacy KVM passthrough and use it when starting VMs and hotplugging
    devices to live machine.

commit f24150b1f53530e0ec057405665b771c776af9ac
Author: Peter Krempa <pkrempa>
Date:   Tue Oct 8 15:47:36 2013 +0200

    qemu: hostdev: Fix function spacing and header formatting

commit 9d13298901f2edb375d54a710480a6bf994cb0e5
Author: Peter Krempa <pkrempa>
Date:   Thu Sep 19 16:48:23 2013 +0200

    qemu: hostdev: Refactor PCI passhrough handling
    
    To simplify future patches dealing with this code, simplify and refactor
    some conditions to switch statements.

A small fixup is needed too:

commit 9c228e08175bdd76372c236298ce129d9a64e0d8
Author: Michal Privoznik <mprivozn>
Date:   Thu Oct 10 12:32:49 2013 +0200

    qemu: Init @pcidevs in qemuPrepareHostdevPCIDevices
    
    At the beginning of the function qemuPrepareHostdevPCICheckSupport() is
    called. After that @pcidevs is initialized. However, if the very first
    command fails, we go to 'cleanup' label where virObjectUnref(pcidevs) is
    called. Obviously, it is called before @pcidevs was able to get
    initialized. Compiler warns about it:
    
      CC       qemu/libvirt_driver_qemu_impl_la-qemu_hostdev.lo
    qemu/qemu_hostdev.c: In function 'qemuPrepareHostdevPCIDevices':
    qemu/qemu_hostdev.c:824:19: error: 'pcidevs' may be used uninitialized in this function [-Werror=maybe-uninitialized]
         virObjectUnref(pcidevs);
                       ^
    cc1: all warnings being treated as errors

Hi, Peter

There are many commit in comment 1, comment 4 and comment 5. Verified with scenario 1 and 2 with build libvirt-1.1.1-10.el7.x86_64, not sure whether it is engough, would you please help to check whether need add other test scenarios? Thanks in advanced.

Scenario1:
Here is the verification steps of the commit in comment 1:
1. prepare one env, that the iommu_group contains 2 network card.
# lspci |grep 82576
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
# readlink -f /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/
/sys/kernel/iommu_groups/14
# ls /sys/kernel/iommu_groups/14/devices/
0000:03:00.0  0000:03:00.1

2. add one network card to the guest with vfio driver, note the managed="yes".
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

3. didn't change the driver of another network card, the guest will failed to boot up with error.
# virsh start rhel7
error: Failed to start domain rhel7
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized

4. edit the dumpxml of this guest, change managed from "yes" to "no"
<hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
5. start the guest again.
# virsh start rhel7
error: Failed to start domain rhel7
error: Unable to allow access for device path /dev/vfio/14: No such file or directory

6. change the driver of another PCI device (which is not assigned in the guest, 0000:03:00.1)
# virsh nodedev-detach pci_0000_03_00_1 --driver vfio
Device pci_0000_03_00_1 detached
# readlink -f /sys/bus/pci/devices/0000\:03\:00.1/driver/
/sys/bus/pci/drivers/vfio-pci

7. start the guest 3rd time.
# virsh start rhel7
error: Failed to start domain rhel7
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized



Scenario2:
As for the commit in step4, libvirt will choose the vfio driver for the pci devices first, the verification steps is like following, didn't specify the driver here.
1. add the following xml to the guest:
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

2. start the guest:
# virsh start a
error: Failed to start domain a
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device pci-assign,configfd=25,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Failed to assign device "hostdev0" : Operation not permitted
qemu-kvm: -device pci-assign,configfd=25,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed.
qemu-kvm: -device pci-assign,configfd=25,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 'kvm-pci-assign' could not be initialized

3. set allow_unsafe_assignment value to 1:
# echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts 
[root@sriov1 ~]# cat /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts 
Y

4. start the guest again
# virsh start a
Domain a started

5. check the qemu command, the assigned pci driver is not vfio now, since the vfio module didn't be probed now.
qemu     17986     1 99 16:44 ?        00:00:11 /usr/libexec/qemu-kvm -name a
...............
-device pci-assign,configfd=25,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

6. destroy the guest, probe the vfio_pci module:
modprobe vfio_pci
[root@sriov1 ~]# lsmod |grep vfio
vfio_pci               32197  0 
vfio_iommu_type1       17677  0 
vfio                   20055  2 vfio_iommu_type1,vfio_pci

7. start the guest again, it seems the guest is assigned the PCI with VFIO driver after probe the vfio module
# virsh start a
error: Failed to start domain a
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to set iommu for container: Operation not permitted
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to setup container for group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized

8. change the other pci devices to vfio, which is in the same iommu group.
# virsh nodedev-detach pci_0000_03_00_1 --driver vfio
Device pci_0000_03_00_1 detached

9. set the value of all_unsafe_interrupts in vfio_iommu_type1 module to 1
# echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
[root@sriov1 ~]# cat /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 
Y

10. start the guest again
# virsh start a
Domain a started

11. check the dumpxml of the guest, libvirt will assign the vfio driver to the assigned pci first, if the vfio module is probed in the host.
# virsh dumpxml a
......
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
......

12. check the qemu command, the pci is assigned with vfio
# ps -ef |grep qemu
qemu     18375     1 33 16:51 ?        00:00:40 /usr/libexec/qemu-kvm -name a 
......
-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

Also needinfo laine, 
Since the patches in this bug is too much, would you please help to check whether the verification steps in comment 8 in enough? If I missed some one, please let me know kindly? thanks.

I think that we shouldn't be relying on the qemu error output in steps 3 and 7 of scenario 1. Once we've detected that the vfio driver is explicitly requested, but not loaded, we should log an appropriate error of our own, without starting up qemu.

hi, Laine,

I agree with you, the error message in step 3 and 7 seems get from the qemu logs, it is too long and not clearly.
I check with the libvirt build libvirt-1.1.1-12.el7.x86_64, the error message in step 3 and 7 is still like that, no change. Will we improve that error message recently?

(In reply to Laine Stump from comment #10)
> I think that we shouldn't be relying on the qemu error output in steps 3 and
> 7 of scenario 1. Once we've detected that the vfio driver is explicitly
> requested, but not loaded, we should log an appropriate error of our own,
> without starting up qemu.

I just noticed that the default for virsh nodedev-detach (and virNodeDeviceDetachFlags()) is to bind to pci-stub (which is used for kvm device assignment) rather than  This needs to be changed to match the behavior of the device assignment itself, i.e.:

1) if no driver is specified, bind to vfio-pci if available, else bind to pci-stub if available, else error

2) if kvm is specified in flags, bind to pci-stub if available, else error

3) if vfio is specified in flags, bind to vfio-pci if available, else error.

(and to answer your question, I believe whatever is in the current RHEL7 build is the same state as upstream).

Forgot to mention that I opened Bug 1031588 to address the problem I noted in Comment 12. This bug should not be held up from verification due to that detail.

There is a little typo, not bug 1031588, should be bug 1035188.

(In reply to Laine Stump from comment #13)
> Forgot to mention that I opened Bug 1031588 to address the problem I noted
> in Comment 12. This bug should not be held up from verification due to that
> detail.

Since there is another bug 1035188 to trace the issue in comment 12, so change this bug status to verify.

hi, Laine,

Find one issue, if didn't specify the vfio driver in the vf xml, it will be failed to hot-plug one VF to the guest.

Since this bug is in "Verified" status, can I open a new bug to track this issue?

Version:
libvirt-1.1.1-16.el7.x86_64
qemu-kvm-1.5.3-30.el7.x86_64
3.10.0-64.el7.x86_64

Steps:
1. prepare one xml like the following one, didn't specify the vfio driver in the xml.
# cat vf-hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address bus='0x03' slot='0x10' function='0x0'/>
  </source>
</hostdev>

2. detach the pci driver to vfio:
# virsh nodedev-detach pci_0000_03_10_0
Device pci_0000_03_10_0 detached

3. hot-plug the vf to one running guest:
# virsh attach-device a vf-hostdev.xml 
error: Failed to attach device from vf-hostdev.xml
error: internal error: unable to execute QEMU command 'getfd': No file descriptor supplied via SCM_RIGHTS

4. check the audit.log info
# ausearch -m avc
time->Thu Dec 26 14:31:09 2013
type=SYSCALL msg=audit(1388039469.295:796): arch=c000003e syscall=47 success=yes exit=1 a0=15 a1=7fffa774d9f0 a2=40000000 a3=0 items=0 ppid=1 pid=4424 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107 egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm" exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c43,c266 key=(null)
type=AVC msg=audit(1388039469.295:796): avc:  denied  { write } for  pid=4424 comm="qemu-kvm" path="/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.0/config" dev="sysfs" ino=18231 scontext=system_u:system_r:svirt_t:s0:c43,c266 tcontext=system_u:object_r:sysfs_t:s0 tclass=file

File another bug to track the issue in this comments:
https://bugzilla.redhat.com/show_bug.cgi?id=1046598

(In reply to Zhang Xuesong from comment #16)
> hi, Laine,
> 
> Find one issue, if didn't specify the vfio driver in the vf xml, it will be
> failed to hot-plug one VF to the guest.
> 
> Since this bug is in "Verified" status, can I open a new bug to track this
> issue?
> 
> Version:
> libvirt-1.1.1-16.el7.x86_64
> qemu-kvm-1.5.3-30.el7.x86_64
> 3.10.0-64.el7.x86_64
> 
> Steps:
> 1. prepare one xml like the following one, didn't specify the vfio driver in
> the xml.
> # cat vf-hostdev.xml 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>   <source>
>     <address bus='0x03' slot='0x10' function='0x0'/>
>   </source>
> </hostdev>
> 
> 2. detach the pci driver to vfio:
> # virsh nodedev-detach pci_0000_03_10_0
> Device pci_0000_03_10_0 detached
> 
> 3. hot-plug the vf to one running guest:
> # virsh attach-device a vf-hostdev.xml 
> error: Failed to attach device from vf-hostdev.xml
> error: internal error: unable to execute QEMU command 'getfd': No file
> descriptor supplied via SCM_RIGHTS
> 
> 4. check the audit.log info
> # ausearch -m avc
> time->Thu Dec 26 14:31:09 2013
> type=SYSCALL msg=audit(1388039469.295:796): arch=c000003e syscall=47
> success=yes exit=1 a0=15 a1=7fffa774d9f0 a2=40000000 a3=0 items=0 ppid=1
> pid=4424 auid=4294967295 uid=107 gid=107 euid=107 suid=107 fsuid=107
> egid=107 sgid=107 fsgid=107 tty=(none) ses=4294967295 comm="qemu-kvm"
> exe="/usr/libexec/qemu-kvm" subj=system_u:system_r:svirt_t:s0:c43,c266
> key=(null)
> type=AVC msg=audit(1388039469.295:796): avc:  denied  { write } for 
> pid=4424 comm="qemu-kvm"
> path="/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.0/config" dev="sysfs"
> ino=18231 scontext=system_u:system_r:svirt_t:s0:c43,c266
> tcontext=system_u:object_r:sysfs_t:s0 tclass=file

Verify this bug with latest libvirt build libvirt-1.1.1-23.el7.x86_64, the steps is same with the ones in comment 8. Some result is changed, but as expected, since there are some related changes after this bug verified, such as the vfio module will be loaded automatically while hot-plug or assign.

Scenario 1:
1. prepare one env, that the iommu_group contains 2 network card.
# lspci |grep 82576
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
# readlink -f /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/
/sys/kernel/iommu_groups/14
# ls /sys/kernel/iommu_groups/14/devices/
0000:03:00.0  0000:03:00.1

2. add one network card to the guest with vfio driver, note the managed="yes".
<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

3. didn't change the driver of another network card, the guest will failed to boot up with error.
# virsh start b
error: Failed to start domain b
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized


4. edit the dumpxml of this guest, change managed from "yes" to "no"
<hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
5. start the guest again.
# virsh start b
error: Failed to start domain b
error: Path '/dev/vfio/14' is not accessible: No such file or directory


6. change the driver of another PCI device (which is not assigned in the guest, 0000:03:00.1)
# virsh nodedev-detach pci_0000_03_00_1
Device pci_0000_03_00_1 detached
# readlink -f /sys/bus/pci/devices/0000\:03\:00.1/driver/
/sys/bus/pci/drivers/vfio-pci

7. start the guest 3rd time.
# virsh start b
error: Failed to start domain b
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: error, group 14 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized




Scenario2:
1. add the following xml to the guest:
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

2. start the guest:
# virsh start b
error: Failed to start domain b
error: internal error: early end of file from monitor: possible problem:
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to set iommu for container: Operation not permitted
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to setup container for group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: vfio: failed to get group 14
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device initialization failed.
qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3: Device 'vfio-pci' could not be initialized


3. set allow_unsafe_assignment value to 1:
# echo 1  > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts 

4. start the guest again
# virsh start b
Domain b started

5. check the qemu command, the assigned pci driver is vfio.
qemu     27522     1 56 14:05 ?        00:00:36  /usr/libexec/qemu-kvm -name b
...............
-device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6


6. check the dumpxml of the guest.
# virsh dumpxml b|grep hostdev -A 5
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.