Bug 984112 - libvirt does not check for successful device_del
libvirt does not check for successful device_del
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.0
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jiri Denemark
Virtualization Bugs
:
Depends On:
Blocks: 971313
  Show dependency treegraph
 
Reported: 2013-07-12 16:40 EDT by Jiri Denemark
Modified: 2014-06-17 20:52 EDT (History)
23 users (show)

See Also:
Fixed In Version: libvirt-1.1.1-1.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 807023
Environment:
Last Closed: 2014-06-13 08:35:01 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
vm auto shut off after detaching device during booting. (5.76 KB, text/plain)
2013-07-31 22:50 EDT, Hu Jianwei
no flags Details

  None (edit)
Description Jiri Denemark 2013-07-12 16:40:44 EDT
+++ This bug was initially created as a clone of Bug #807023 +++

Description of problem:
libvirt appears to always assume a device_del is successful.  This is not always the case.  Issuing a device_del simply registers a request to eject the device with the guest, it doesn't guarantee removal.  There are numerous reasons a guest may fail to release the device, however, when this happens libvirt does not report error and continues as if the removal was successful.

In my case I see this with a tg3 devices assigned to the guest.  The guest driver attempts to do a power state transition on the device, which doesn't work, and gets stuck.  The guest never calls the eject method for the device, but libvirt continues to unbind the device from pci-stub and pretend the removal was successful.

We need to come up with some strategy of libvirt polling the devices attached to the guest after the device_del, checking whether the removal was successful.

Version-Release number of selected component (if applicable):
libvirt-1.1.0-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Assign a tg3 device to a guest
2. Bring up the interface in the guest
3. Try to remove the device
  
Actual results:
libvirt reports success and unbinds the device from pci-stub, meanwhile the guest still owns the device (/proc/pid/fd still reports all the device file descriptors open).

Expected results:
libvirt checks for device removal and reports error after some timeout and doesn't unbind device from pci-stub.

Additional info:
Comment 1 Jiri Denemark 2013-07-16 14:35:04 EDT
First round of patches pushed upstream:

commit 89b7bb75d7ef03edd2f127449f0f0ed2bf79e4fe
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Tue Jul 9 22:30:57 2013 +0200

    qemu: Add qemuDomainReleaseDeviceAddress to remove any address

commit a22ae222eeb4a764d6d791a06c6368e38a41d720
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Tue Jul 9 23:43:22 2013 +0200

    qemu: Separate disk device removal into a standalone function

commit 92758a71d803e64ed565bd3f102c18d5b41a436c
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Jul 10 00:10:32 2013 +0200

    qemu: Separate controller removal into a standalone function

commit ac68a785ccccd35f0edb75b9118c653c4c4cbeb0
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Jul 10 00:37:12 2013 +0200

    qemu: Separate net device removal into a standalone function

commit 53f3739afea44fc74b64cf18e6e35bc593bc7711
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Thu Jul 11 11:14:16 2013 +0200

    qemu: Separate host device removal into a standalone function
Comment 2 Laine Stump 2013-07-17 11:03:17 EDT
I think that making the device removal cleanup asynchronous will also fix Bug 971313, which is caused by libvirt relabelling the device's sysfs files before qemu is finished with them.
Comment 3 Jiri Denemark 2013-07-18 10:03:40 EDT
Another round of patches pushed upstream:

commit d077cda4e9f9e551f5cad6fcd2c3ed666bd138aa
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Tue Jul 16 21:16:09 2013 +0200

    qemu: Separate char device removal into a standalone function

commit 4421e257dd29ccf5144984d902819828c4b90268
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Jun 19 15:27:29 2013 +0200

    Add VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED event

commit ee023f6c1fd7d356aa66d41499b1a37ce0180eb0
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Jun 19 15:28:00 2013 +0200

    examples: Handle VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED event

commit 1f71de59f67ee7dcd4957b61d35bde52211a6c29
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Wed Jun 19 15:48:55 2013 +0200

    Clarify virDomainDetachDeviceFlags documentation

commit ae951e724c3371ee9b118a369c3be683f2325547
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Thu Jul 11 16:54:16 2013 +0200

    Add virDomainDefFindDevice for looking up a device by its alias

commit ab47cc9bf97d838a92d8b4dd40bd83c8e50e1083
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Thu Jul 11 17:07:26 2013 +0200

    qemu: Add support for DEVICE_DELETED event

commit 3fbf78bdf331d8ab0a015c7fec885a8e4192f0fa
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Thu Jul 11 17:11:02 2013 +0200

    qemu: Remove devices only after DEVICE_DELETED event

commit 984c01ba5c6bf105263f99ff242c4df15a422e24
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Thu Jul 11 17:30:56 2013 +0200

    qemu: Emit VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED events
Comment 4 Jiri Denemark 2013-07-18 10:06:03 EDT
The only thing which remains now is to handle devices that finished removal while libvirtd was not running. That is, libvirtd has to check what devices disappeared when it's reconnection to domains.
Comment 5 Jiri Denemark 2013-07-22 07:15:56 EDT
This is now fully implemented upstream as of v1.1.0-278-g0dfb8a1 with the following patches:

commit 09e9f28118af848cb0a40a99a5b1502e0de88495
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Fri Jul 19 14:55:41 2013 +0200

    util: Non-existent string array does not contain any string
    
    Make virStringArrayHasString return false when called on a non-existent
    string array.

commit d327ac53282abc91ad563375bcb89333ac5bc31b
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Fri Jul 19 14:59:36 2013 +0200

    conf: Make error reporting in virDomainDefFindDevice optional

commit 58b147ad07c9432b53e66ca20aff74d812647c57
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Fri Jul 19 15:01:38 2013 +0200

    qemu: Introduce qemuMonitorGetDeviceAliases
    
    This API provides a NULL-terminated list of devices which are currently
    attached to a QEMU domain.

commit 0dfb8a1b9e7fd3c4bb85e0487a83726f08167654
Author: Jiri Denemark <jdenemar@redhat.com>
Date:   Fri Jul 19 15:08:29 2013 +0200

    qemu: Unplug devices that disappeared when libvirtd was down
    
    In case libvirtd is asked to unplug a device but the device is actually
    unplugged later when libvirtd is not running, we need to detect that and
    remove such device when libvirtd starts again and reconnects to running
    domains.
Comment 6 Hu Jianwei 2013-07-31 07:55:08 EDT
Hi Jiri,

I verified this bug according to comment 5 of 807023, https://bugzilla.redhat.com/show_bug.cgi?id=807023#c5. But I still reproduced it with those steps.

1. Define a guest with the following hostdev device
[root@SRIOV2 ~]# virsh dumpxml r6| grep -A 10 hostdev
...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
...

2.Start the guest, at the same time detach that device in another terminal.
[root@SRIOV2 ~]# virsh start r7_g
Domain r7_g started
[root@SRIOV2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 7     r7_g                           running

In another terminal:
[root@SRIOV2 ~]# virsh detach-device r7_g hostdev.xml
Device detached successfully

3. In guest, we can still find that pci device.

4. Check the fd info.
[root@SRIOV2 ~]# ll /proc/`pidof qemu-kvm`/fd | grep "/sys/devices/"
lrwx------. 1 qemu qemu 64 Jul 31 07:36 23 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource0
lrwx------. 1 qemu qemu 64 Jul 31 07:36 24 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource2
lrwx------. 1 qemu qemu 64 Jul 31 07:36 25 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/config
lrwx------. 1 qemu qemu 64 Jul 31 07:36 26 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource3


I don't know whether this is right steps to validate this bug or not. If my reproducible steps had some mistake, please show me the correct steps to test it. Thanks in advance.
Comment 7 Jiri Denemark 2013-07-31 08:11:00 EDT
The important thing is whether the hostdev can still be found in domain xml. That is, whether repeating step 1 gives you the hostdev xml. If yes, then everything works as expected (previously libvirt would remove the device from domain xml even though it was still present in the guest). In any case, could you file a separate bug requesting virsh to be more cleare about what successful result of device-detach means?
Comment 8 Hu Jianwei 2013-07-31 22:48:18 EDT
As your said, I tried to reproduce this bug with my comment 6 using libvirt-1.1.0-1.el7.x86_64(the bug was reported on it.) But I failed because the domain will shut off automatically due to "qemu: hardware error: pci write failed, ret = -1 errno = 13". Domian log has been attached.

Also, I re-checked domain xml, that hostdev still existed using libvirt-1.1.1-1.el7.x86_64, details as below.

Version:
[root@SRIOV2 ~]# rpm -q libvirt qemu-kvm kernel
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
kernel-3.10.0-3.el7.x86_64

In guest:
[root@test005 ~]# lspci | grep Eth
00:03.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

In domain xml:
[root@SRIOV2 ~]# virsh dumpxml r7_g | grep -A 10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...


Based on my above testing results, I think my reproducible steps must have some wrong(I can not reproduce it with those steps using old libvirt version). Could you share to me another right method, if possible?

BR,
Jianwei
Comment 9 Hu Jianwei 2013-07-31 22:50:08 EDT
Created attachment 781412 [details]
vm auto shut off after detaching device during booting.
Comment 10 Laine Stump 2013-08-01 11:18:03 EDT
That is exactly the error in Bug 971313. Can you verify:

1) that there is a related AVC in /var/log/audit/audit.log

2) that it works properly after running "setenforce 0"

The fact that this error still occurs makes me believe that either qemu is sending the "device is finally really deleted" event too soon, or that libvirt is relabelling the sysfs nodes for the device before qemu sends that event. (hopefully the latter, since we will be able to fix that).
Comment 11 Jiri Denemark 2013-08-01 12:22:38 EDT
IIUC it's not hapenning anymore but happened with an older libvirt-1.1.0-1.el7, which is OK. That's one of several symptoms of the bug... libvirt used to remove (and relabel) the device even though the guest or qemu were still using it. Which now doesn't happen and the device remains plugged from the libvirt's POV.
Comment 12 Hu Jianwei 2013-08-01 21:48:35 EDT
(In reply to Laine Stump from comment #10)
> That is exactly the error in Bug 971313. Can you verify:
> 
> 1) that there is a related AVC in /var/log/audit/audit.log
> 
> 2) that it works properly after running "setenforce 0"
> 
> The fact that this error still occurs makes me believe that either qemu is
> sending the "device is finally really deleted" event too soon, or that
> libvirt is relabelling the sysfs nodes for the device before qemu sends that
> event. (hopefully the latter, since we will be able to fix that).

Below bug will block my further test for your mentioned above.We can not assign a vf to domain by now.
The SRIOV host will be rebooted by the kernel panic while attach the 82576 vf card to the guest, https://bugzilla.redhat.com/show_bug.cgi?id=990987
Comment 13 Hu Jianwei 2013-08-06 06:14:38 EDT
This bug can not reproduce on libvirt-1.1.1-1.el7.x86_64 with below scenario.

[root@SRIOV2 ~]# rpm -q libvirt qemu-kvm kernel
libvirt-1.1.1-1.el7.x86_64
qemu-kvm-1.5.2-1.el7.x86_64
kernel-3.10.0-3.el7.x86_64

1. Define a guest with the following hostdev device
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
    </hostdev>

2. Start the guest, at the same time detach that device in another terminal.
[root@SRIOV2 ~]# virsh start r7_g
Domain r7_g started
[root@SRIOV2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 7     r7_g                           running

In another terminal,execute below command immediately:
[root@SRIOV2 ~]# virsh detach-device r7_g hostdev.xml
Device detached successfully

3. Dumpxml domain xml and check process fd.
[root@SRIOV2 ~]# virsh dumpxml r7_g | grep -A10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>

[root@SRIOV2 ~]# virsh dumpxml r7_g --inactive| grep -A10 hostdev
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>

[root@SRIOV2 ~]# ll /proc/`pidof qemu-kvm`/fd | grep "/sys/devices/"
lrwx------. 1 qemu qemu 64 Aug  6 05:57 23 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/config
lrwx------. 1 qemu qemu 64 Aug  6 05:57 24 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource0
lrwx------. 1 qemu qemu 64 Aug  6 05:57 25 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource2
lrwx------. 1 qemu qemu 64 Aug  6 05:57 26 -> /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/resource3

4. In guest, to check the nic status, it's in the domain.
[root@test005 ~]# lspci| grep Eth
00:03.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
[root@test005 ~]# ifconfig eth1
eth1      Link encap:Ethernet  HWaddr B4:B5:2F:AF:A1:00  
          inet addr:10.66.71.177  Bcast:10.66.71.255  Mask:255.255.254.0
          inet6 addr: fe80::b6b5:2fff:feaf:a100/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:10 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1932 (1.8 KiB)  TX bytes:1588 (1.5 KiB)
          Interrupt:11 Memory:fc000000-fc020000

[root@test005 ~]# ping 10.66.7.130
PING 10.66.7.130 (10.66.7.130) 56(84) bytes of data.
64 bytes from 10.66.7.130: icmp_seq=1 ttl=62 time=0.324 ms
64 bytes from 10.66.7.130: icmp_seq=2 ttl=62 time=0.282 ms
64 bytes from 10.66.7.130: icmp_seq=3 ttl=62 time=0.402 ms

--- 10.66.7.130 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2391ms
rtt min/avg/max/mdev = 0.282/0.336/0.402/0.049 ms
[root@test005 ~]#

5. check audit.log, no AVC related logs.




For patch "qemu: Unplug devices that disappeared when libvirtd was down"

I plan to test below scenario:

1. Using my above steps, fail to detach hostdev;
2. To stop libvirtd;(After that, all libvirt related commands will be disabled.)
3. Using qemu command line to detach hostdev or others;
4. To start libvirtd;
5. check the domain xml.

By now, I did not find a good way to detach this pci device from qemu-kvm process.
Comment 14 Hu Jianwei 2013-08-06 06:21:03 EDT
According to comment 7, filed a new bug to track that issue.
https://bugzilla.redhat.com/show_bug.cgi?id=993631
Comment 15 Hu Jianwei 2013-08-27 05:45:22 EDT
> For patch "qemu: Unplug devices that disappeared when libvirtd was down"
> 
> I plan to test below scenario:
> 
> 1. Using my above steps, fail to detach hostdev;
> 2. To stop libvirtd;(After that, all libvirt related commands will be
> disabled.)
> 3. Using qemu command line to detach hostdev or others;
> 4. To start libvirtd;
> 5. check the domain xml.
> 
> By now, I did not find a good way to detach this pci device from qemu-kvm
> process.

According to comment 13, new scenario, added need_info.
Comment 17 Hu Jianwei 2013-10-28 01:44:52 EDT
(In reply to Hu Jianwei from comment #15)
> > For patch "qemu: Unplug devices that disappeared when libvirtd was down"
> > 
> > I plan to test below scenario:
> > 
> > 1. Using my above steps, fail to detach hostdev;
> > 2. To stop libvirtd;(After that, all libvirt related commands will be
> > disabled.)
> > 3. Using qemu command line to detach hostdev or others;
> > 4. To start libvirtd;
> > 5. check the domain xml.
> > 
> > By now, I did not find a good way to detach this pci device from qemu-kvm
> > process.
> 
> According to comment 13, new scenario, added need_info.

Is there any ideas for that scenario? Thanks.
Comment 18 Jiri Denemark 2013-11-01 06:20:49 EDT
I think the following scenario could work:

1. start a domain but leave it in grub menu
2. attach the PCI device
3. detach it (it won't actually detach the device since guest OS is not responding; check domain XML it's still there
4. kill libvirtd
5. let the guest OS boot
6. connect to qemu monitor manually and tell it to remove the PCI device
7. start libvirtd
8. check domain XML

There might be an easier way to do this, though.
Comment 19 Hu Jianwei 2013-11-05 22:47:15 EST
Verified another scenario in comment 18, it's passed.

Version:
libvirt-1.1.1-11.el7.x86_64
qemu-kvm-1.5.3-10.el7.x86_64
kernel-3.10.0-33.el7.x86_64

1. Start domian with hostdev device.
[root@sriov2 ~]# virsh start r7
Domain r7 started
 
[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...

2. Leave OS in grub menu, and detach that hostdev devcie.(it won't actually detach the device since guest OS is not responding; check domain XML it's still there)
[root@sriov2 ~]# cat hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
</hostdev>

[root@sriov2 ~]#  virsh detach-device r7 hostdev.xml
Device detached successfully

[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
...

3. Stop libvirtd service.
[root@sriov2 ~]# service libvirtd stop
Redirecting to /bin/systemctl stop  libvirtd.service
[root@sriov2 ~]# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: inactive (dead) since Wed 2013-11-06 11:19:00 CST; 1s ago
  Process: 1768 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=exited, status=0/SUCCESS)
 Main PID: 1768 (code=exited, status=0/SUCCESS)

Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 17
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 18
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 19
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 20
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 21
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 22
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 25
Nov 06 11:17:40 sriov2 libvirtd[1768]: 2013-11-06 03:17:40.179+0000: 3490: debug : virFileClose:90 : Closed fd 28
Nov 06 11:19:00 sriov2 systemd[1]: Stopping Virtualization daemon...
Nov 06 11:19:00 sriov2 systemd[1]: Stopped Virtualization daemon.
[root@sriov2 ~]# 

4.Continue to boot domain, and check the interface in domain, it's in domain.
(in guest)
[root@localhost ~]# lspci| grep Eth
00:03.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
[root@localhost ~]# ip ad show ens3
3: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:1b:21:55:b3:bc brd ff:ff:ff:ff:ff:ff
    inet 10.66.71.137/23 brd 10.66.71.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::21b:21ff:fe55:b3bc/64 scope link 
       valid_lft forever preferred_lft forever
[root@localhost ~]#

5. Connect to qemu monitor manually and remove the PCI device
[root@sriov2 ~]# ps aux | grep r7 | sed "s/-device/\n-device/g"
...
-device pci-assign,configfd=24,host=10:00.0,id=hostdev0,bus=pci.0,addr=0x3 
...

[root@sriov2 ~]# socat stdin unix-connect:/var/lib/libvirt/qemu/r7.monitor
{"QMP": {"version": {"qemu": {"micro": 3, "minor": 5, "major": 1}, "package": " (qemu-kvm-1.5.3-10.el7)"}, "capabilities": []}}
{"execute":"qmp_capabilities","id":"libvirt-1"}
{"return": {}, "id": "libvirt-1"}
{"execute":"device_del","arguments":{"id":"hostdev0"}}
{"return": {}}
{"timestamp": {"seconds": 1383708178, "microseconds": 95727}, "event": "DEVICE_DELETED", "data": {"device": "hostdev0", "path": "/machine/peripheral/hostdev0"}}
^C[root@sriov2 ~]# 

6. Start libvirtd service.
[root@sriov2 ~]# service libvirtd start
Redirecting to /bin/systemctl start  libvirtd.service
[root@sriov2 ~]# service libvirtd status
Redirecting to /bin/systemctl status  libvirtd.service
libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
   Active: active (running) since Wed 2013-11-06 11:23:43 CST; 2s ago
 Main PID: 3606 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─2888 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
           ├─3606 /usr/sbin/libvirtd
           └─3645 /usr/bin/python /usr/bin/firewall-cmd --direct --passthrough ipv4 --table filter --delete INPUT --in-interface virbr0 --protocol tcp --destinatio...

Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 7
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 8
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 9
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 10
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 11
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 12
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 13
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 14
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 16
Nov 06 11:23:45 sriov2 libvirtd[3606]: 2013-11-06 03:23:45.650+0000: 3645: debug : virFileClose:90 : Closed fd 18

7. Check domain XML, libvirtd updated the domain xml automatically after reconnecting to running domain.
[root@sriov2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     r7                             running

[root@sriov2 ~]# virsh dumpxml r7| grep hostdev -aA 8
            <=== No output.

We can get expected results in two scenarios(in comment 13 and 18), so changed to verified.
Comment 20 Xuesong Zhang 2014-02-20 05:22:45 EST
In the latest libvirt build libvirt-1.1.1-23.el7.x86_64, this bug is verified.
But one of the expected result is changed like following, since the default pci driver is vfio on rhel7 now.

changed the expected result of step 3 in comment 13:
# ll /proc/`pidof qemu-kvm`/fd | grep vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 21 -> /dev/vfio/84
lrwx------. 1 qemu qemu 64 Feb 20 04:54 22 -> /dev/vfio/vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 23 -> anon_inode:kvm-vfio
lrwx------. 1 qemu qemu 64 Feb 20 04:54 24 -> anon_inode:[vfio-device]
Comment 21 Ludek Smid 2014-06-13 08:35:01 EDT
This request was resolved in Red Hat Enterprise Linux 7.0.

Contact your manager or support representative in case you have further questions about the request.

Note You need to log in before you can comment on or make changes to this bug.