Bug 984112

First round of patches pushed upstream:

commit 89b7bb75d7ef03edd2f127449f0f0ed2bf79e4fe
Author: Jiri Denemark <jdenemar>
Date:   Tue Jul 9 22:30:57 2013 +0200

    qemu: Add qemuDomainReleaseDeviceAddress to remove any address

commit a22ae222eeb4a764d6d791a06c6368e38a41d720
Author: Jiri Denemark <jdenemar>
Date:   Tue Jul 9 23:43:22 2013 +0200

    qemu: Separate disk device removal into a standalone function

commit 92758a71d803e64ed565bd3f102c18d5b41a436c
Author: Jiri Denemark <jdenemar>
Date:   Wed Jul 10 00:10:32 2013 +0200

    qemu: Separate controller removal into a standalone function

commit ac68a785ccccd35f0edb75b9118c653c4c4cbeb0
Author: Jiri Denemark <jdenemar>
Date:   Wed Jul 10 00:37:12 2013 +0200

    qemu: Separate net device removal into a standalone function

commit 53f3739afea44fc74b64cf18e6e35bc593bc7711
Author: Jiri Denemark <jdenemar>
Date:   Thu Jul 11 11:14:16 2013 +0200

    qemu: Separate host device removal into a standalone function

I think that making the device removal cleanup asynchronous will also fix Bug 971313, which is caused by libvirt relabelling the device's sysfs files before qemu is finished with them.

Another round of patches pushed upstream:

commit d077cda4e9f9e551f5cad6fcd2c3ed666bd138aa
Author: Jiri Denemark <jdenemar>
Date:   Tue Jul 16 21:16:09 2013 +0200

    qemu: Separate char device removal into a standalone function

commit 4421e257dd29ccf5144984d902819828c4b90268
Author: Jiri Denemark <jdenemar>
Date:   Wed Jun 19 15:27:29 2013 +0200

    Add VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED event

commit ee023f6c1fd7d356aa66d41499b1a37ce0180eb0
Author: Jiri Denemark <jdenemar>
Date:   Wed Jun 19 15:28:00 2013 +0200

    examples: Handle VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED event

commit 1f71de59f67ee7dcd4957b61d35bde52211a6c29
Author: Jiri Denemark <jdenemar>
Date:   Wed Jun 19 15:48:55 2013 +0200

    Clarify virDomainDetachDeviceFlags documentation

commit ae951e724c3371ee9b118a369c3be683f2325547
Author: Jiri Denemark <jdenemar>
Date:   Thu Jul 11 16:54:16 2013 +0200

    Add virDomainDefFindDevice for looking up a device by its alias

commit ab47cc9bf97d838a92d8b4dd40bd83c8e50e1083
Author: Jiri Denemark <jdenemar>
Date:   Thu Jul 11 17:07:26 2013 +0200

    qemu: Add support for DEVICE_DELETED event

commit 3fbf78bdf331d8ab0a015c7fec885a8e4192f0fa
Author: Jiri Denemark <jdenemar>
Date:   Thu Jul 11 17:11:02 2013 +0200

    qemu: Remove devices only after DEVICE_DELETED event

commit 984c01ba5c6bf105263f99ff242c4df15a422e24
Author: Jiri Denemark <jdenemar>
Date:   Thu Jul 11 17:30:56 2013 +0200

    qemu: Emit VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED events

The only thing which remains now is to handle devices that finished removal while libvirtd was not running. That is, libvirtd has to check what devices disappeared when it's reconnection to domains.

This is now fully implemented upstream as of v1.1.0-278-g0dfb8a1 with the following patches:

commit 09e9f28118af848cb0a40a99a5b1502e0de88495
Author: Jiri Denemark <jdenemar>
Date:   Fri Jul 19 14:55:41 2013 +0200

    util: Non-existent string array does not contain any string
    
    Make virStringArrayHasString return false when called on a non-existent
    string array.

commit d327ac53282abc91ad563375bcb89333ac5bc31b
Author: Jiri Denemark <jdenemar>
Date:   Fri Jul 19 14:59:36 2013 +0200

    conf: Make error reporting in virDomainDefFindDevice optional

commit 58b147ad07c9432b53e66ca20aff74d812647c57
Author: Jiri Denemark <jdenemar>
Date:   Fri Jul 19 15:01:38 2013 +0200

    qemu: Introduce qemuMonitorGetDeviceAliases
    
    This API provides a NULL-terminated list of devices which are currently
    attached to a QEMU domain.

commit 0dfb8a1b9e7fd3c4bb85e0487a83726f08167654
Author: Jiri Denemark <jdenemar>
Date:   Fri Jul 19 15:08:29 2013 +0200

    qemu: Unplug devices that disappeared when libvirtd was down
    
    In case libvirtd is asked to unplug a device but the device is actually
    unplugged later when libvirtd is not running, we need to detect that and
    remove such device when libvirtd starts again and reconnects to running
    domains.

Hi Jiri,

I verified this bug according to comment 5 of 807023, https://bugzilla.redhat.com/show_bug.cgi?id=807023#c5. But I still reproduced it with those steps.

1. Define a guest with the following hostdev device
[root@SRIOV2 ~]# virsh dumpxml r6| grep -A 10 hostdev
...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
...

2.Start the guest, at the same time detach that device in another terminal.
[root@SRIOV2 ~]# virsh start r7_g
Domain r7_g started
[root@SRIOV2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 7     r7_g                           running

In another terminal:
[root@SRIOV2 ~]# virsh detach-device r7_g hostdev.xml
Device detached successfully

3. In guest, we can still find that pci device.

4. Check the fd info.
[root@SRIOV2 ~]# ll /proc/`pidof qemu-kvm`/fd | grep "/sys/devices/"
lrwx------. 1 qemu qemu 64 Jul 31 07:36 23 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource0
lrwx------. 1 qemu qemu 64 Jul 31 07:36 24 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource2
lrwx------. 1 qemu qemu 64 Jul 31 07:36 25 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/config
lrwx------. 1 qemu qemu 64 Jul 31 07:36 26 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource3


I don't know whether this is right steps to validate this bug or not. If my reproducible steps had some mistake, please show me the correct steps to test it. Thanks in advance.

The important thing is whether the hostdev can still be found in domain xml. That is, whether repeating step 1 gives you the hostdev xml. If yes, then everything works as expected (previously libvirt would remove the device from domain xml even though it was still present in the guest). In any case, could you file a separate bug requesting virsh to be more cleare about what successful result of device-detach means?

As your said, I tried to reproduce this bug with my comment 6 using libvirt-1.1.0-1.el7.x86_64(the bug was reported on it.) But I failed because the domain will shut off automatically due to "qemu: hardware error: pci write failed, ret = -1 errno = 13". Domian log has been attached.

Also, I re-checked domain xml, that hostdev still existed using libvirt-1.1.1-1.el7.x86_64, details as below.

Version:
[root@SRIOV2 ~]# rpm -q libvirt qemu-kvm kernel
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
kernel-3.10.0-3.el7.x86_64

In guest:
[root@test005 ~]# lspci | grep Eth
00:03.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

In domain xml:
[root@SRIOV2 ~]# virsh dumpxml r7_g | grep -A 10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...


Based on my above testing results, I think my reproducible steps must have some wrong(I can not reproduce it with those steps using old libvirt version). Could you share to me another right method, if possible?

BR,
Jianwei

Created attachment 781412 [details]
vm auto shut off after detaching device during booting.

That is exactly the error in Bug 971313. Can you verify:

1) that there is a related AVC in /var/log/audit/audit.log

2) that it works properly after running "setenforce 0"

The fact that this error still occurs makes me believe that either qemu is sending the "device is finally really deleted" event too soon, or that libvirt is relabelling the sysfs nodes for the device before qemu sends that event. (hopefully the latter, since we will be able to fix that).

IIUC it's not hapenning anymore but happened with an older libvirt-1.1.0-1.el7, which is OK. That's one of several symptoms of the bug... libvirt used to remove (and relabel) the device even though the guest or qemu were still using it. Which now doesn't happen and the device remains plugged from the libvirt's POV.

(In reply to Laine Stump from comment #10)
> That is exactly the error in Bug 971313. Can you verify:
> 
> 1) that there is a related AVC in /var/log/audit/audit.log
> 
> 2) that it works properly after running "setenforce 0"
> 
> The fact that this error still occurs makes me believe that either qemu is
> sending the "device is finally really deleted" event too soon, or that
> libvirt is relabelling the sysfs nodes for the device before qemu sends that
> event. (hopefully the latter, since we will be able to fix that).

Below bug will block my further test for your mentioned above.We can not assign a vf to domain by now.
The SRIOV host will be rebooted by the kernel panic while attach the 82576 vf card to the guest, https://bugzilla.redhat.com/show_bug.cgi?id=990987

This bug can not reproduce on libvirt-1.1.1-1.el7.x86_64 with below scenario.

[root@SRIOV2 ~]# rpm -q libvirt qemu-kvm kernel
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
kernel-3.10.0-3.el7.x86_64

1. Define a guest with the following hostdev device
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

2. Start the guest, at the same time detach that device in another terminal.
[root@SRIOV2 ~]# virsh start r7_g
Domain r7_g started
[root@SRIOV2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 7     r7_g                           running

In another terminal，execute below command immediately:
[root@SRIOV2 ~]# virsh detach-device r7_g hostdev.xml
Device detached successfully

3. Dumpxml domain xml and check process fd.
[root@SRIOV2 ~]# virsh dumpxml r7_g | grep -A10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>

[root@SRIOV2 ~]# virsh dumpxml r7_g --inactive| grep -A10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>

[root@SRIOV2 ~]# ll /proc/`pidof qemu-kvm`/fd | grep "/sys/devices/"
lrwx------. 1 qemu qemu 64 Aug  6 05:57 23 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/config
lrwx------. 1 qemu qemu 64 Aug  6 05:57 24 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource0
lrwx------. 1 qemu qemu 64 Aug  6 05:57 25 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource2
lrwx------. 1 qemu qemu 64 Aug  6 05:57 26 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource3

4. In guest, to check the nic status, it's in the domain.
[root@test005 ~]# lspci| grep Eth
00:03.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
[root@test005 ~]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr B4:B5:2F:AF:A1:00  
          inet addr:10.66.71.177  Bcast:10.66.71.255  Mask:255.255.254.0
          inet6 addr: fe80::b6b5:2fff:feaf:a100/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1932 (1.8 KiB)  TX bytes:1588 (1.5 KiB)
          Interrupt:11 Memory:fc000000-fc020000

[root@test005 ~]# ping 10.66.7.130
PING 10.66.7.130 (10.66.7.130) 56(84) bytes of data.
64 bytes from 10.66.7.130: icmp_seq=1 ttl=62 time=0.324 ms
64 bytes from 10.66.7.130: icmp_seq=2 ttl=62 time=0.282 ms
64 bytes from 10.66.7.130: icmp_seq=3 ttl=62 time=0.402 ms

--- 10.66.7.130 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2391ms
rtt min/avg/max/mdev = 0.282/0.336/0.402/0.049 ms
[root@test005 ~]#

5. check audit.log, no AVC related logs.




For patch "qemu: Unplug devices that disappeared when libvirtd was down"

I plan to test below scenario:

1. Using my above steps, fail to detach hostdev;
2. To stop libvirtd;(After that, all libvirt related commands will be disabled.)
3. Using qemu command line to detach hostdev or others;
4. To start libvirtd;
5. check the domain xml.

By now, I did not find a good way to detach this pci device from qemu-kvm process.

According to comment 7, filed a new bug to track that issue.
https://bugzilla.redhat.com/show_bug.cgi?id=993631

> For patch "qemu: Unplug devices that disappeared when libvirtd was down"
> 
> I plan to test below scenario:
> 
> 1. Using my above steps, fail to detach hostdev;
> 2. To stop libvirtd;(After that, all libvirt related commands will be
> disabled.)
> 3. Using qemu command line to detach hostdev or others;
> 4. To start libvirtd;
> 5. check the domain xml.
> 
> By now, I did not find a good way to detach this pci device from qemu-kvm
> process.

According to comment 13, new scenario, added need_info.

(In reply to Hu Jianwei from comment #15)
> > For patch "qemu: Unplug devices that disappeared when libvirtd was down"
> > 
> > I plan to test below scenario:
> > 
> > 1. Using my above steps, fail to detach hostdev;
> > 2. To stop libvirtd;(After that, all libvirt related commands will be
> > disabled.)
> > 3. Using qemu command line to detach hostdev or others;
> > 4. To start libvirtd;
> > 5. check the domain xml.
> > 
> > By now, I did not find a good way to detach this pci device from qemu-kvm
> > process.
> 
> According to comment 13, new scenario, added need_info.

Is there any ideas for that scenario? Thanks.

I think the following scenario could work:

1. start a domain but leave it in grub menu
2. attach the PCI device
3. detach it (it won't actually detach the device since guest OS is not responding; check domain XML it's still there
4. kill libvirtd
5. let the guest OS boot
6. connect to qemu monitor manually and tell it to remove the PCI device
7. start libvirtd
8. check domain XML

There might be an easier way to do this, though.

Verified another scenario in comment 18, it's passed.

Version:
libvirt-1.1.1-11.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64
kernel-3.10.0-33.el7.x86_64

1. Start domian with hostdev device.
[root@sriov2 ~]# virsh start r7
Domain r7 started
 
[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...

2. Leave OS in grub menu, and detach that hostdev devcie.(it won't actually detach the device since guest OS is not responding; check domain XML it's still there)
[root@sriov2 ~]# cat hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
</hostdev>

[root@sriov2 ~]#  virsh detach-device r7 hostdev.xml
Device detached successfully

[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...

3. Stop libvirtd service.
[root@sriov2 ~]# service libvirtd stop
Redirecting to /bin/systemctl stop  libvirtd.service
[root@sriov2 ~]# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: inactive (dead) since Wed 2013-11-06 11:19:00 CST; 1s ago
  Process: 1768 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 1768 (code=exited, status=0/SUCCESS)

Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 17
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 18
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 19
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 20
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 21
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 22
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 25
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 28
Nov 06 11:19:00 sriov2 systemd[1]: Stopping Virtualization daemon...
Nov 06 11:19:00 sriov2 systemd[1]: Stopped Virtualization daemon.
[root@sriov2 ~]# 

4.Continue to boot domain, and check the interface in domain, it's in domain.
(in guest)
[root@localhost ~]# lspci| grep Eth
00:03.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
[root@localhost ~]# ip ad show ens3
3: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:55:b3:bc brd ff:ff:ff:ff:ff:ff
    inet 10.66.71.137/23 brd 10.66.71.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::21b:21ff:fe55:b3bc/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

5. Connect to qemu monitor manually and remove the PCI device
[root@sriov2 ~]# ps aux | grep r7 | sed "s/-device/\n-device/g"
...
-device pci-assign,configfd=24,host=10:00.0,id=hostdev0,bus=pci.0,addr=0x3 
...

[root@sriov2 ~]# socat stdin unix-connect:/var/lib/libvirt/qemu/r7.monitor
{"QMP": {"version": {"qemu": {"micro": 3, "minor": 5, "major": 1}, "package": " (qemu-kvm-1.5.3-10.el7)"}, "capabilities": []}}
{"execute":"qmp_capabilities","id":"libvirt-1"}
{"return": {}, "id": "libvirt-1"}
{"execute":"device_del","arguments":{"id":"hostdev0"}}
{"return": {}}
{"timestamp": {"seconds": 1383708178, "microseconds": 95727}, "event": "DEVICE_DELETED", "data": {"device": "hostdev0", "path": "/machine/peripheral/hostdev0"}}
^C[root@sriov2 ~]# 

6. Start libvirtd service.
[root@sriov2 ~]# service libvirtd start
Redirecting to /bin/systemctl start  libvirtd.service
[root@sriov2 ~]# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Wed 2013-11-06 11:23:43 CST; 2s ago
 Main PID: 3606 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─2888 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
           ├─3606 /usr/sbin/libvirtd
           └─3645 /usr/bin/python /usr/bin/firewall-cmd --direct --passthrough ipv4 --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destinatio...

Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 7
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 8
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 9
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 10
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 11
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 12
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 13
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 14
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 16
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 18

7. Check domain XML, libvirtd updated the domain xml automatically after reconnecting to running domain.
[root@sriov2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     r7                             running

[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
            <=== No output.

We can get expected results in two scenarios(in comment 13 and 18), so changed to verified.

In the latest libvirt build libvirt-1.1.1-23.el7.x86_64, this bug is verified.
But one of the expected result is changed like following, since the default pci driver is vfio on rhel7 now.

changed the expected result of step 3 in comment 13:
# ll /proc/`pidof qemu-kvm`/fd | grep vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 21 -> /dev/vfio/84
lrwx------. 1 qemu qemu 64 Feb 20 04:54 22 -> /dev/vfio/vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 23 -> anon_inode:kvm-vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 24 -> anon_inode:[vfio-device]

This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.