Bug 843016 - PCI device can't be returned to host automatically after hot-unplugging 'hostdev' type network interface with managed='yes'
PCI device can't be returned to host automatically after hot-unplugging 'host...
Status: CLOSED DUPLICATE of bug 813752
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.4
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Osier Yang
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-25 06:21 EDT by Alex Jia
Modified: 2012-09-25 05:35 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-25 05:35:15 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alex Jia 2012-07-25 06:21:17 EDT
Description of problem:
The network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device.

Version-Release number of selected component (if applicable):
# rpm -q libvirt qemu-kvm-rhev
libvirt-0.9.13-3.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64

How reproducible:
always

Steps to Reproduce:

# lspci|grep 82576
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)


# modprobe -r kvm_intel kvm
# modprobe kvm allow_unsafe_assigned_interrupts=1
# modprobe kvm_intel

# virsh start myVF
Domain myVF started

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/igbvf

# cat hostdev.xml 
    <interface type='hostdev' managed='yes'>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
    </interface>

# virsh attach-device myVF hostdev.xml 
Device attached successfully

# virsh dumpxml myVF
<domain type='kvm' id='9'>
  <name>myVF</name>
  ......
  <devices>
  ......
    <interface type='network'>
      <mac address='52:54:00:81:94:f7'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:ec:52:44'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
  ......
  </devices>
  ......
</domain>

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/pci-stub

# virsh detach-device myVF hostdev.xml 
error: Failed to detach device from hostdev.xml
error: operation failed: network device 52:54:00:a4:8b:17 not found

Notes, there are 2 questions in here, the one is libvirt allows to use 'virsh attach-device' to attach a interface XML to guest, but don't allow detach it from guest via 'virsh detach-device'. the other is the error information also is weird, I haven't specified any MAC address in hostdev.xml, it seems 'virsh detach-device' default to detach the first network interface.


# virsh detach-interface myVF hostdev 52:54:00:ec:52:44
Interface detached successfully

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver

Notes, the network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device.

# virsh destroy myVF
Domain myVF destroyed

# virsh nodedev-reattach pci_0000_03_10_1
Device pci_0000_03_10_1 re-attached

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/bus/pci/drivers/igbvf


Notes, I must manually run 'virsh nodedev-reattach' to explicity return the PCI device to host, I think managed='yes' doesn't make sense in here.


Actual results:

# readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
/sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver

Notes, the network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device.


Expected results:
libvirt should return the host PCI device to host with managed='yes' mode after hot-unplugging PCI device.

Additional info:
Comment 2 Osier Yang 2012-07-30 04:09:28 EDT
(In reply to comment #0)
> Description of problem:
> The network interface with managed='yes' mode, but libvirt hasn't return the
> host PCI device to host after hot-unplugging PCI device.
> 
> Version-Release number of selected component (if applicable):
> # rpm -q libvirt qemu-kvm-rhev
> libvirt-0.9.13-3.el6.x86_64
> qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64
> 
> How reproducible:
> always
> 
> Steps to Reproduce:
> 
> # lspci|grep 82576
> 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
> 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network
> Connection (rev 01)
> 03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev
> 01)
> 
> 
> # modprobe -r kvm_intel kvm
> # modprobe kvm allow_unsafe_assigned_interrupts=1
> # modprobe kvm_intel
> 
> # virsh start myVF
> Domain myVF started
> 
> # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
> /sys/bus/pci/drivers/igbvf
> 
> # cat hostdev.xml 
>     <interface type='hostdev' managed='yes'>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> function='0x1'/>
>       </source>
>     </interface>
> 
> # virsh attach-device myVF hostdev.xml 
> Device attached successfully
> 
> # virsh dumpxml myVF
> <domain type='kvm' id='9'>
>   <name>myVF</name>
>   ......
>   <devices>
>   ......
>     <interface type='network'>
>       <mac address='52:54:00:81:94:f7'/>
>       <source network='default'/>
>       <target dev='vnet0'/>
>       <model type='virtio'/>
>       <alias name='net0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>     </interface>
>     <interface type='hostdev' managed='yes'>
>       <mac address='52:54:00:ec:52:44'/>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> function='0x1'/>
>       </source>
>       <alias name='hostdev0'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
>     </interface>
>   ......
>   </devices>
>   ......
> </domain>
> 
> # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
> /sys/bus/pci/drivers/pci-stub
> 
> # virsh detach-device myVF hostdev.xml 
> error: Failed to detach device from hostdev.xml
> error: operation failed: network device 52:54:00:a4:8b:17 not found
> 
> Notes, there are 2 questions in here, the one is libvirt allows to use
> 'virsh attach-device' to attach a interface XML to guest, but don't allow
> detach it from guest via 'virsh detach-device'. 

It's allowed. Just a bug.

the other is the error
> information also is weird, I haven't specified any MAC address in
> hostdev.xml, it seems 'virsh detach-device' default to detach the first
> network interface.
> 

I think your hostdev.xml is same as what in the domain config. But to be confirmed, What's the hostdev.xml? I'm afraid of this is bug since commit 3b1c191fe.

> 
> # virsh detach-interface myVF hostdev 52:54:00:ec:52:44
> Interface detached successfully
> 
> # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
> /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver

It's a another bug. Given that this bug title is about "managed=yes". Please
file a separate bug for the previous one.
Comment 3 Alex Jia 2012-07-30 04:19:37 EDT
(In reply to comment #2)
> > Notes, there are 2 questions in here, the one is libvirt allows to use
> > 'virsh attach-device' to attach a interface XML to guest, but don't allow
> > detach it from guest via 'virsh detach-device'. 
> 
> It's allowed. Just a bug.
> 
> the other is the error
> > information also is weird, I haven't specified any MAC address in
> > hostdev.xml, it seems 'virsh detach-device' default to detach the first
> > network interface.
> > 
> 
> I think your hostdev.xml is same as what in the domain config. But to be
> confirmed, What's the hostdev.xml? I'm afraid of this is bug since commit
> 3b1c191fe.

Yeah, the hostdev.xml is the same to previous:

> 
> > 
> > # virsh detach-interface myVF hostdev 52:54:00:ec:52:44
> > Interface detached successfully
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/
> > /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver
> 
> It's a another bug. Given that this bug title is about "managed=yes". Please
> file a separate bug for the previous one.

Yeah, this bug is major about 'managed=yes', I raised 2 questions in the bug,  I will file a separate bug for 'virsh detach-device' issue then paste bug id in here, thanks for your comment.
Comment 4 Alex Jia 2012-07-30 04:30:22 EDT
(In reply to comment #3)
> > It's a another bug. Given that this bug title is about "managed=yes". Please
> > file a separate bug for the previous one.
> 
> Yeah, this bug is major about 'managed=yes', I raised 2 questions in the
> bug,  I will file a separate bug for 'virsh detach-device' issue then paste
> bug id in here, thanks for your comment.

File a new bug 844287 for this question.
Comment 5 Alex Jia 2012-08-31 03:25:02 EDT
(In reply to comment #0)
> Description of problem:
> The network interface with managed='yes' mode, but libvirt hasn't return the
> host PCI device to host after hot-unplugging PCI device.
> 
> 
> # cat hostdev.xml 
>     <interface type='hostdev' managed='yes'>
>       <source>
>         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> function='0x1'/>
>       </source>
>     </interface>
> 

BTW, if I replace "<hostdev mode='subsystem' type='pci' managed='yes'> ... </hostdev>" XML configuration with above hostdev.xml, I will get the same result. I clearly remembered it's okay ago, but not sure which version libvirt is, it should be a regression bug, this issue also can be found on libvirt-0.9.10-21.el6.x86_64 (RHEL6.3 release version).
Comment 6 Osier Yang 2012-08-31 08:05:06 EDT
(In reply to comment #5)
> (In reply to comment #0)
> > Description of problem:
> > The network interface with managed='yes' mode, but libvirt hasn't return the
> > host PCI device to host after hot-unplugging PCI device.
> > 
> > 
> > # cat hostdev.xml 
> >     <interface type='hostdev' managed='yes'>
> >       <source>
> >         <address type='pci' domain='0x0000' bus='0x03' slot='0x10'
> > function='0x1'/>
> >       </source>
> >     </interface>
> > 
> 
> BTW, if I replace "<hostdev mode='subsystem' type='pci' managed='yes'> ...
> </hostdev>" XML configuration with above hostdev.xml, I will get the same
> result. I clearly remembered it's okay ago, but not sure which version
> libvirt is, it should be a regression bug,

If the problem exists for general host pci device. That's a regression bug,
otherwise not.
 
 this issue also can be found on
> libvirt-0.9.10-21.el6.x86_64 (RHEL6.3 release version).

It looks to me all the attach and detach codes are good. Could you check libvirt
log to see if there is some error. And /proc/iomem to see if qemu-kvm is still cleaning up the device by:

cat /proc/iomem | grep kvm_assigned_device

If it matches the device, then it's a known problem of qemu.
Comment 7 Alex Jia 2012-09-03 06:52:44 EDT
(In reply to comment #6)
> 
> If the problem exists for general host pci device. That's a regression bug,
> otherwise not.

Yeah, we can meet this question for a normal PCI device.
 
> cat /proc/iomem | grep kvm_assigned_device

Nothing output for me, and will double confirm it tomrrow.  

> If it matches the device, then it's a known problem of qemu.

And Kyla also can reproduce this issue.
Comment 8 Alex Jia 2012-09-24 04:44:39 EDT
# virsh list
 Id    Name                           State
----------------------------------------------------
 111   myRHEL6                        running

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/e1000e

# cat /proc/iomem | grep kvm
# cat /tmp/hostdev.xml 
<hostdev mode='subsystem' type='pci' managed='yes'>
   <source>
      <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
   </source>
</hostdev>

# virsh attach-device myRHEL6 /tmp/hostdev.xml 
Device attached successfully

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/pci-stub

# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device

# virsh detach-device myRHEL6 /tmp/hostdev.xml 
Device detached successfully

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver

# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device

Notes, the device can't be returned to host. 

# echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver

# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device

Notes, still can't reattach the device to host.

# virsh nodedev-reattach pci_0000_00_19_0
Device pci_0000_00_19_0 re-attached

Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via virsh.

# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device

Notes, the same result to previous.

# virsh destroy myRHEL6
Domain myRHEL6 destroyed

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver

# cat /proc/iomem | grep kvm

Notes, nothing is outputted by above cmdline, it should be okay.

# echo "0000:00:19.0" > /sys/bus/pci/drivers_probe

Notes, manually reprobe the device driver.

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/e1000e

Notes, it's okay now.
Comment 9 Alex Jia 2012-09-24 05:11:10 EDT
(In reply to comment #8)
> # virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  111   myRHEL6                        running
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/bus/pci/drivers/e1000e
> 
> # cat /proc/iomem | grep kvm
> # cat /tmp/hostdev.xml 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>    <source>
>       <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
>    </source>
> </hostdev>
> 
> # virsh attach-device myRHEL6 /tmp/hostdev.xml 
> Device attached successfully
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/bus/pci/drivers/pci-stub
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> # virsh detach-device myRHEL6 /tmp/hostdev.xml 
> Device detached successfully
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> Notes, the device can't be returned to host. 
> 
> # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> Notes, still can't reattach the device to host.
> 
> # virsh nodedev-reattach pci_0000_00_19_0
> Device pci_0000_00_19_0 re-attached

# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver
# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device
Comment 10 Alex Jia 2012-09-24 05:14:33 EDT
BTW, if I repeatly run test steps(virsh {at,de}tach-device) of comment 8, everything is okay, the device can be returned to host after virsh detach-device, but if I change a new guest then do these again, I can reproduce it again.
Comment 11 Osier Yang 2012-09-24 08:30:44 EDT
(In reply to comment #9)
> (In reply to comment #8)
> > # virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> >  111   myRHEL6                        running
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/bus/pci/drivers/e1000e
> > 
> > # cat /proc/iomem | grep kvm
> > # cat /tmp/hostdev.xml 
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> >    <source>
> >       <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
> >    </source>
> > </hostdev>
> > 
> > # virsh attach-device myRHEL6 /tmp/hostdev.xml 
> > Device attached successfully
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/bus/pci/drivers/pci-stub
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > # virsh detach-device myRHEL6 /tmp/hostdev.xml 
> > Device detached successfully
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/devices/pci0000:00/0000:00:19.0/driver
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > Notes, the device can't be returned to host. 
> > 
> > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/devices/pci0000:00/0000:00:19.0/driver
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > Notes, still can't reattach the device to host.
> > 
> > # virsh nodedev-reattach pci_0000_00_19_0
> > Device pci_0000_00_19_0 re-attached
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device

Note that nodedev-reattach force to unbind + remove_id + reprobe, and it fails to reprobe the driver even so.
Comment 12 Osier Yang 2012-09-24 08:32:01 EDT
(In reply to comment #10)
> BTW, if I repeatly run test steps(virsh {at,de}tach-device) of comment 8,
> everything is okay, the device can be returned to host after virsh
> detach-device, but if I change a new guest then do these again, I can
> reproduce it again.

The successful cases proved libvirt logic is right.
Comment 13 Osier Yang 2012-09-24 08:35:49 EDT
(In reply to comment #8)
> # virsh list
>  Id    Name                           State
> ----------------------------------------------------
>  111   myRHEL6                        running
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/bus/pci/drivers/e1000e
> 
> # cat /proc/iomem | grep kvm
> # cat /tmp/hostdev.xml 
> <hostdev mode='subsystem' type='pci' managed='yes'>
>    <source>
>       <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
>    </source>
> </hostdev>
> 
> # virsh attach-device myRHEL6 /tmp/hostdev.xml 
> Device attached successfully
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/bus/pci/drivers/pci-stub
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> # virsh detach-device myRHEL6 /tmp/hostdev.xml 
> Device detached successfully
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> Notes, the device can't be returned to host. 
> 
> # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> Notes, still can't reattach the device to host.
> 
> # virsh nodedev-reattach pci_0000_00_19_0
> Device pci_0000_00_19_0 re-attached
> 
> Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via
> virsh.
> 
> # cat /proc/iomem | grep kvm
>       febdb000-febdbfff : kvm_assigned_device
>       febe0000-febfffff : kvm_assigned_device
> 
> Notes, the same result to previous.
> 

<******************>
> # virsh destroy myRHEL6
> Domain myRHEL6 destroyed
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/devices/pci0000:00/0000:00:19.0/driver
> 
> # cat /proc/iomem | grep kvm
> 
> Notes, nothing is outputted by above cmdline, it should be okay.
> 
> # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> 
> Notes, manually reprobe the device driver.
> 
> # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> /sys/bus/pci/drivers/e1000e
> 
> Notes, it's okay now.
</***************>

This success, and failure in #c9 (nodedev-reattach) shows the reprobing
failure relates with qemu-kvm doesn't do the cleanup for the device timely.
Comment 14 Osier Yang 2012-09-24 08:37:02 EDT
(In reply to comment #13)
> (In reply to comment #8)
> > # virsh list
> >  Id    Name                           State
> > ----------------------------------------------------
> >  111   myRHEL6                        running
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/bus/pci/drivers/e1000e
> > 
> > # cat /proc/iomem | grep kvm
> > # cat /tmp/hostdev.xml 
> > <hostdev mode='subsystem' type='pci' managed='yes'>
> >    <source>
> >       <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
> >    </source>
> > </hostdev>
> > 
> > # virsh attach-device myRHEL6 /tmp/hostdev.xml 
> > Device attached successfully
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/bus/pci/drivers/pci-stub
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > # virsh detach-device myRHEL6 /tmp/hostdev.xml 
> > Device detached successfully
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/devices/pci0000:00/0000:00:19.0/driver
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > Notes, the device can't be returned to host. 
> > 
> > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/devices/pci0000:00/0000:00:19.0/driver
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device
> > 
> > Notes, still can't reattach the device to host.
> > 
> > # virsh nodedev-reattach pci_0000_00_19_0
> > Device pci_0000_00_19_0 re-attached
> > 
> > Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via
> > virsh.
> > 
> > # cat /proc/iomem | grep kvm
> >       febdb000-febdbfff : kvm_assigned_device
> >       febe0000-febfffff : kvm_assigned_device

This should be the key point.

> > 
> > Notes, the same result to previous.
> > 
> 
> <******************>
> > # virsh destroy myRHEL6
> > Domain myRHEL6 destroyed
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/devices/pci0000:00/0000:00:19.0/driver
> > 
> > # cat /proc/iomem | grep kvm
> > 
> > Notes, nothing is outputted by above cmdline, it should be okay.
> > 
> > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> > 
> > Notes, manually reprobe the device driver.
> > 
> > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > /sys/bus/pci/drivers/e1000e
> > 
> > Notes, it's okay now.
> </***************>
> 
> This success, and failure in #c9 (nodedev-reattach) shows the reprobing
> failure relates with qemu-kvm doesn't do the cleanup for the device timely.
Comment 15 Osier Yang 2012-09-24 08:57:47 EDT
(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #8)
> > > # virsh list
> > >  Id    Name                           State
> > > ----------------------------------------------------
> > >  111   myRHEL6                        running
> > > 
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/bus/pci/drivers/e1000e
> > > 
> > > # cat /proc/iomem | grep kvm
> > > # cat /tmp/hostdev.xml 
> > > <hostdev mode='subsystem' type='pci' managed='yes'>
> > >    <source>
> > >       <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
> > >    </source>
> > > </hostdev>
> > > 
> > > # virsh attach-device myRHEL6 /tmp/hostdev.xml 
> > > Device attached successfully
> > > 
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/bus/pci/drivers/pci-stub
> > > 
> > > # cat /proc/iomem | grep kvm
> > >       febdb000-febdbfff : kvm_assigned_device
> > >       febe0000-febfffff : kvm_assigned_device
> > > 
> > > # virsh detach-device myRHEL6 /tmp/hostdev.xml 
> > > Device detached successfully
> > > 
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/devices/pci0000:00/0000:00:19.0/driver
> > > 
> > > # cat /proc/iomem | grep kvm
> > >       febdb000-febdbfff : kvm_assigned_device
> > >       febe0000-febfffff : kvm_assigned_device
> > > 
> > > Notes, the device can't be returned to host. 
> > > 
> > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/devices/pci0000:00/0000:00:19.0/driver
> > > 
> > > # cat /proc/iomem | grep kvm
> > >       febdb000-febdbfff : kvm_assigned_device
> > >       febe0000-febfffff : kvm_assigned_device
> > > 
> > > Notes, still can't reattach the device to host.
> > > 
> > > # virsh nodedev-reattach pci_0000_00_19_0
> > > Device pci_0000_00_19_0 re-attached
> > > 
> > > Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via
> > > virsh.
> > > 
> > > # cat /proc/iomem | grep kvm
> > >       febdb000-febdbfff : kvm_assigned_device
> > >       febe0000-febfffff : kvm_assigned_device
> 
> This should be the key point.

And libvirt already waits enough time (100 * 100 * 1000 ms) for qemu-kvm's cleanup.

> 
> > > 
> > > Notes, the same result to previous.
> > > 
> > 
> > <******************>
> > > # virsh destroy myRHEL6
> > > Domain myRHEL6 destroyed
> > > 
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/devices/pci0000:00/0000:00:19.0/driver
> > > 
> > > # cat /proc/iomem | grep kvm
> > > 
> > > Notes, nothing is outputted by above cmdline, it should be okay.
> > > 
> > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
> > > 
> > > Notes, manually reprobe the device driver.
> > > 
> > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
> > > /sys/bus/pci/drivers/e1000e
> > > 
> > > Notes, it's okay now.
> > </***************>
> > 
> > This success, and failure in #c9 (nodedev-reattach) shows the reprobing
> > failure relates with qemu-kvm doesn't do the cleanup for the device timely.
Comment 16 Osier Yang 2012-09-25 05:35:15 EDT
Okay, the root problem here is libvirt assumes device_del always succeeds, which
is not true. This needs both work from qemu and libvirt to make sure the device
is really hot unplugged from the guest. For this bug, the device should be
actually not unplugged from guest successfully, and libvirt assumes it does,
and moves forward to unbind the device, and reprobe the drivers. And this
answers why the PCI BARs mapping is not cleaned up:

# cat /proc/iomem | grep kvm
      febdb000-febdbfff : kvm_assigned_device
      febe0000-febfffff : kvm_assigned_device


See following bugs for more details:

https://bugzilla.redhat.com/show_bug.cgi?id=807023
https://bugzilla.redhat.com/show_bug.cgi?id=813752

And qemu bug:

https://bugzilla.redhat.com/show_bug.cgi?id=813748

So close this as duplicate with 813752

*** This bug has been marked as a duplicate of bug 813752 ***

Note You need to log in before you can comment on or make changes to this bug.