Bug 843016
| Summary: | PCI device can't be returned to host automatically after hot-unplugging 'hostdev' type network interface with managed='yes' | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Alex Jia <ajia> |
| Component: | libvirt | Assignee: | Osier Yang <jyang> |
| Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.4 | CC: | acathrow, dallan, dyasny, dyuan, mzhan |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-09-25 09:35:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
(In reply to comment #0) > Description of problem: > The network interface with managed='yes' mode, but libvirt hasn't return the > host PCI device to host after hot-unplugging PCI device. > > Version-Release number of selected component (if applicable): > # rpm -q libvirt qemu-kvm-rhev > libvirt-0.9.13-3.el6.x86_64 > qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64 > > How reproducible: > always > > Steps to Reproduce: > > # lspci|grep 82576 > 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network > Connection (rev 01) > 03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > 03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev > 01) > > > # modprobe -r kvm_intel kvm > # modprobe kvm allow_unsafe_assigned_interrupts=1 > # modprobe kvm_intel > > # virsh start myVF > Domain myVF started > > # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ > /sys/bus/pci/drivers/igbvf > > # cat hostdev.xml > <interface type='hostdev' managed='yes'> > <source> > <address type='pci' domain='0x0000' bus='0x03' slot='0x10' > function='0x1'/> > </source> > </interface> > > # virsh attach-device myVF hostdev.xml > Device attached successfully > > # virsh dumpxml myVF > <domain type='kvm' id='9'> > <name>myVF</name> > ...... > <devices> > ...... > <interface type='network'> > <mac address='52:54:00:81:94:f7'/> > <source network='default'/> > <target dev='vnet0'/> > <model type='virtio'/> > <alias name='net0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x03' > function='0x0'/> > </interface> > <interface type='hostdev' managed='yes'> > <mac address='52:54:00:ec:52:44'/> > <source> > <address type='pci' domain='0x0000' bus='0x03' slot='0x10' > function='0x1'/> > </source> > <alias name='hostdev0'/> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' > function='0x0'/> > </interface> > ...... > </devices> > ...... > </domain> > > # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ > /sys/bus/pci/drivers/pci-stub > > # virsh detach-device myVF hostdev.xml > error: Failed to detach device from hostdev.xml > error: operation failed: network device 52:54:00:a4:8b:17 not found > > Notes, there are 2 questions in here, the one is libvirt allows to use > 'virsh attach-device' to attach a interface XML to guest, but don't allow > detach it from guest via 'virsh detach-device'. It's allowed. Just a bug. the other is the error > information also is weird, I haven't specified any MAC address in > hostdev.xml, it seems 'virsh detach-device' default to detach the first > network interface. > I think your hostdev.xml is same as what in the domain config. But to be confirmed, What's the hostdev.xml? I'm afraid of this is bug since commit 3b1c191fe. > > # virsh detach-interface myVF hostdev 52:54:00:ec:52:44 > Interface detached successfully > > # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ > /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver It's a another bug. Given that this bug title is about "managed=yes". Please file a separate bug for the previous one. (In reply to comment #2) > > Notes, there are 2 questions in here, the one is libvirt allows to use > > 'virsh attach-device' to attach a interface XML to guest, but don't allow > > detach it from guest via 'virsh detach-device'. > > It's allowed. Just a bug. > > the other is the error > > information also is weird, I haven't specified any MAC address in > > hostdev.xml, it seems 'virsh detach-device' default to detach the first > > network interface. > > > > I think your hostdev.xml is same as what in the domain config. But to be > confirmed, What's the hostdev.xml? I'm afraid of this is bug since commit > 3b1c191fe. Yeah, the hostdev.xml is the same to previous: > > > > > # virsh detach-interface myVF hostdev 52:54:00:ec:52:44 > > Interface detached successfully > > > > # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ > > /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver > > It's a another bug. Given that this bug title is about "managed=yes". Please > file a separate bug for the previous one. Yeah, this bug is major about 'managed=yes', I raised 2 questions in the bug, I will file a separate bug for 'virsh detach-device' issue then paste bug id in here, thanks for your comment. (In reply to comment #3) > > It's a another bug. Given that this bug title is about "managed=yes". Please > > file a separate bug for the previous one. > > Yeah, this bug is major about 'managed=yes', I raised 2 questions in the > bug, I will file a separate bug for 'virsh detach-device' issue then paste > bug id in here, thanks for your comment. File a new bug 844287 for this question. (In reply to comment #0) > Description of problem: > The network interface with managed='yes' mode, but libvirt hasn't return the > host PCI device to host after hot-unplugging PCI device. > > > # cat hostdev.xml > <interface type='hostdev' managed='yes'> > <source> > <address type='pci' domain='0x0000' bus='0x03' slot='0x10' > function='0x1'/> > </source> > </interface> > BTW, if I replace "<hostdev mode='subsystem' type='pci' managed='yes'> ... </hostdev>" XML configuration with above hostdev.xml, I will get the same result. I clearly remembered it's okay ago, but not sure which version libvirt is, it should be a regression bug, this issue also can be found on libvirt-0.9.10-21.el6.x86_64 (RHEL6.3 release version). (In reply to comment #5) > (In reply to comment #0) > > Description of problem: > > The network interface with managed='yes' mode, but libvirt hasn't return the > > host PCI device to host after hot-unplugging PCI device. > > > > > > # cat hostdev.xml > > <interface type='hostdev' managed='yes'> > > <source> > > <address type='pci' domain='0x0000' bus='0x03' slot='0x10' > > function='0x1'/> > > </source> > > </interface> > > > > BTW, if I replace "<hostdev mode='subsystem' type='pci' managed='yes'> ... > </hostdev>" XML configuration with above hostdev.xml, I will get the same > result. I clearly remembered it's okay ago, but not sure which version > libvirt is, it should be a regression bug, If the problem exists for general host pci device. That's a regression bug, otherwise not. this issue also can be found on > libvirt-0.9.10-21.el6.x86_64 (RHEL6.3 release version). It looks to me all the attach and detach codes are good. Could you check libvirt log to see if there is some error. And /proc/iomem to see if qemu-kvm is still cleaning up the device by: cat /proc/iomem | grep kvm_assigned_device If it matches the device, then it's a known problem of qemu. (In reply to comment #6) > > If the problem exists for general host pci device. That's a regression bug, > otherwise not. Yeah, we can meet this question for a normal PCI device. > cat /proc/iomem | grep kvm_assigned_device Nothing output for me, and will double confirm it tomrrow. > If it matches the device, then it's a known problem of qemu. And Kyla also can reproduce this issue. # virsh list
Id Name State
----------------------------------------------------
111 myRHEL6 running
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/e1000e
# cat /proc/iomem | grep kvm
# cat /tmp/hostdev.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
</source>
</hostdev>
# virsh attach-device myRHEL6 /tmp/hostdev.xml
Device attached successfully
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/pci-stub
# cat /proc/iomem | grep kvm
febdb000-febdbfff : kvm_assigned_device
febe0000-febfffff : kvm_assigned_device
# virsh detach-device myRHEL6 /tmp/hostdev.xml
Device detached successfully
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver
# cat /proc/iomem | grep kvm
febdb000-febdbfff : kvm_assigned_device
febe0000-febfffff : kvm_assigned_device
Notes, the device can't be returned to host.
# echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver
# cat /proc/iomem | grep kvm
febdb000-febdbfff : kvm_assigned_device
febe0000-febfffff : kvm_assigned_device
Notes, still can't reattach the device to host.
# virsh nodedev-reattach pci_0000_00_19_0
Device pci_0000_00_19_0 re-attached
Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via virsh.
# cat /proc/iomem | grep kvm
febdb000-febdbfff : kvm_assigned_device
febe0000-febfffff : kvm_assigned_device
Notes, the same result to previous.
# virsh destroy myRHEL6
Domain myRHEL6 destroyed
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/devices/pci0000:00/0000:00:19.0/driver
# cat /proc/iomem | grep kvm
Notes, nothing is outputted by above cmdline, it should be okay.
# echo "0000:00:19.0" > /sys/bus/pci/drivers_probe
Notes, manually reprobe the device driver.
# readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/
/sys/bus/pci/drivers/e1000e
Notes, it's okay now.
(In reply to comment #8) > # virsh list > Id Name State > ---------------------------------------------------- > 111 myRHEL6 running > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/bus/pci/drivers/e1000e > > # cat /proc/iomem | grep kvm > # cat /tmp/hostdev.xml > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > </source> > </hostdev> > > # virsh attach-device myRHEL6 /tmp/hostdev.xml > Device attached successfully > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/bus/pci/drivers/pci-stub > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > # virsh detach-device myRHEL6 /tmp/hostdev.xml > Device detached successfully > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > Notes, the device can't be returned to host. > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > Notes, still can't reattach the device to host. > > # virsh nodedev-reattach pci_0000_00_19_0 > Device pci_0000_00_19_0 re-attached # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ /sys/devices/pci0000:00/0000:00:19.0/driver # cat /proc/iomem | grep kvm febdb000-febdbfff : kvm_assigned_device febe0000-febfffff : kvm_assigned_device BTW, if I repeatly run test steps(virsh {at,de}tach-device) of comment 8, everything is okay, the device can be returned to host after virsh detach-device, but if I change a new guest then do these again, I can reproduce it again.
(In reply to comment #9) > (In reply to comment #8) > > # virsh list > > Id Name State > > ---------------------------------------------------- > > 111 myRHEL6 running > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/bus/pci/drivers/e1000e > > > > # cat /proc/iomem | grep kvm > > # cat /tmp/hostdev.xml > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > > </source> > > </hostdev> > > > > # virsh attach-device myRHEL6 /tmp/hostdev.xml > > Device attached successfully > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/bus/pci/drivers/pci-stub > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > # virsh detach-device myRHEL6 /tmp/hostdev.xml > > Device detached successfully > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > Notes, the device can't be returned to host. > > > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > Notes, still can't reattach the device to host. > > > > # virsh nodedev-reattach pci_0000_00_19_0 > > Device pci_0000_00_19_0 re-attached > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device Note that nodedev-reattach force to unbind + remove_id + reprobe, and it fails to reprobe the driver even so. (In reply to comment #10) > BTW, if I repeatly run test steps(virsh {at,de}tach-device) of comment 8, > everything is okay, the device can be returned to host after virsh > detach-device, but if I change a new guest then do these again, I can > reproduce it again. The successful cases proved libvirt logic is right. (In reply to comment #8) > # virsh list > Id Name State > ---------------------------------------------------- > 111 myRHEL6 running > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/bus/pci/drivers/e1000e > > # cat /proc/iomem | grep kvm > # cat /tmp/hostdev.xml > <hostdev mode='subsystem' type='pci' managed='yes'> > <source> > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > </source> > </hostdev> > > # virsh attach-device myRHEL6 /tmp/hostdev.xml > Device attached successfully > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/bus/pci/drivers/pci-stub > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > # virsh detach-device myRHEL6 /tmp/hostdev.xml > Device detached successfully > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > Notes, the device can't be returned to host. > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > Notes, still can't reattach the device to host. > > # virsh nodedev-reattach pci_0000_00_19_0 > Device pci_0000_00_19_0 re-attached > > Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via > virsh. > > # cat /proc/iomem | grep kvm > febdb000-febdbfff : kvm_assigned_device > febe0000-febfffff : kvm_assigned_device > > Notes, the same result to previous. > <******************> > # virsh destroy myRHEL6 > Domain myRHEL6 destroyed > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/devices/pci0000:00/0000:00:19.0/driver > > # cat /proc/iomem | grep kvm > > Notes, nothing is outputted by above cmdline, it should be okay. > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > Notes, manually reprobe the device driver. > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > /sys/bus/pci/drivers/e1000e > > Notes, it's okay now. </***************> This success, and failure in #c9 (nodedev-reattach) shows the reprobing failure relates with qemu-kvm doesn't do the cleanup for the device timely. (In reply to comment #13) > (In reply to comment #8) > > # virsh list > > Id Name State > > ---------------------------------------------------- > > 111 myRHEL6 running > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/bus/pci/drivers/e1000e > > > > # cat /proc/iomem | grep kvm > > # cat /tmp/hostdev.xml > > <hostdev mode='subsystem' type='pci' managed='yes'> > > <source> > > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > > </source> > > </hostdev> > > > > # virsh attach-device myRHEL6 /tmp/hostdev.xml > > Device attached successfully > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/bus/pci/drivers/pci-stub > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > # virsh detach-device myRHEL6 /tmp/hostdev.xml > > Device detached successfully > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > Notes, the device can't be returned to host. > > > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device > > > > Notes, still can't reattach the device to host. > > > > # virsh nodedev-reattach pci_0000_00_19_0 > > Device pci_0000_00_19_0 re-attached > > > > Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via > > virsh. > > > > # cat /proc/iomem | grep kvm > > febdb000-febdbfff : kvm_assigned_device > > febe0000-febfffff : kvm_assigned_device This should be the key point. > > > > Notes, the same result to previous. > > > > <******************> > > # virsh destroy myRHEL6 > > Domain myRHEL6 destroyed > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > # cat /proc/iomem | grep kvm > > > > Notes, nothing is outputted by above cmdline, it should be okay. > > > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > > > Notes, manually reprobe the device driver. > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > /sys/bus/pci/drivers/e1000e > > > > Notes, it's okay now. > </***************> > > This success, and failure in #c9 (nodedev-reattach) shows the reprobing > failure relates with qemu-kvm doesn't do the cleanup for the device timely. (In reply to comment #14) > (In reply to comment #13) > > (In reply to comment #8) > > > # virsh list > > > Id Name State > > > ---------------------------------------------------- > > > 111 myRHEL6 running > > > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/bus/pci/drivers/e1000e > > > > > > # cat /proc/iomem | grep kvm > > > # cat /tmp/hostdev.xml > > > <hostdev mode='subsystem' type='pci' managed='yes'> > > > <source> > > > <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/> > > > </source> > > > </hostdev> > > > > > > # virsh attach-device myRHEL6 /tmp/hostdev.xml > > > Device attached successfully > > > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/bus/pci/drivers/pci-stub > > > > > > # cat /proc/iomem | grep kvm > > > febdb000-febdbfff : kvm_assigned_device > > > febe0000-febfffff : kvm_assigned_device > > > > > > # virsh detach-device myRHEL6 /tmp/hostdev.xml > > > Device detached successfully > > > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > > > # cat /proc/iomem | grep kvm > > > febdb000-febdbfff : kvm_assigned_device > > > febe0000-febfffff : kvm_assigned_device > > > > > > Notes, the device can't be returned to host. > > > > > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > > > # cat /proc/iomem | grep kvm > > > febdb000-febdbfff : kvm_assigned_device > > > febe0000-febfffff : kvm_assigned_device > > > > > > Notes, still can't reattach the device to host. > > > > > > # virsh nodedev-reattach pci_0000_00_19_0 > > > Device pci_0000_00_19_0 re-attached > > > > > > Notes, repeat run 'echo "0000:00:19.0" > /sys/bus/pci/drivers_probe' via > > > virsh. > > > > > > # cat /proc/iomem | grep kvm > > > febdb000-febdbfff : kvm_assigned_device > > > febe0000-febfffff : kvm_assigned_device > > This should be the key point. And libvirt already waits enough time (100 * 100 * 1000 ms) for qemu-kvm's cleanup. > > > > > > > Notes, the same result to previous. > > > > > > > <******************> > > > # virsh destroy myRHEL6 > > > Domain myRHEL6 destroyed > > > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/devices/pci0000:00/0000:00:19.0/driver > > > > > > # cat /proc/iomem | grep kvm > > > > > > Notes, nothing is outputted by above cmdline, it should be okay. > > > > > > # echo "0000:00:19.0" > /sys/bus/pci/drivers_probe > > > > > > Notes, manually reprobe the device driver. > > > > > > # readlink -f /sys/bus/pci/devices/0000\:00\:19.0/driver/ > > > /sys/bus/pci/drivers/e1000e > > > > > > Notes, it's okay now. > > </***************> > > > > This success, and failure in #c9 (nodedev-reattach) shows the reprobing > > failure relates with qemu-kvm doesn't do the cleanup for the device timely. Okay, the root problem here is libvirt assumes device_del always succeeds, which
is not true. This needs both work from qemu and libvirt to make sure the device
is really hot unplugged from the guest. For this bug, the device should be
actually not unplugged from guest successfully, and libvirt assumes it does,
and moves forward to unbind the device, and reprobe the drivers. And this
answers why the PCI BARs mapping is not cleaned up:
# cat /proc/iomem | grep kvm
febdb000-febdbfff : kvm_assigned_device
febe0000-febfffff : kvm_assigned_device
See following bugs for more details:
https://bugzilla.redhat.com/show_bug.cgi?id=807023
https://bugzilla.redhat.com/show_bug.cgi?id=813752
And qemu bug:
https://bugzilla.redhat.com/show_bug.cgi?id=813748
So close this as duplicate with 813752
*** This bug has been marked as a duplicate of bug 813752 ***
|
Description of problem: The network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device. Version-Release number of selected component (if applicable): # rpm -q libvirt qemu-kvm-rhev libvirt-0.9.13-3.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.297.el6_3.x86_64 How reproducible: always Steps to Reproduce: # lspci|grep 82576 03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) 03:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) 03:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01) # modprobe -r kvm_intel kvm # modprobe kvm allow_unsafe_assigned_interrupts=1 # modprobe kvm_intel # virsh start myVF Domain myVF started # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ /sys/bus/pci/drivers/igbvf # cat hostdev.xml <interface type='hostdev' managed='yes'> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> </interface> # virsh attach-device myVF hostdev.xml Device attached successfully # virsh dumpxml myVF <domain type='kvm' id='9'> <name>myVF</name> ...... <devices> ...... <interface type='network'> <mac address='52:54:00:81:94:f7'/> <source network='default'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <interface type='hostdev' managed='yes'> <mac address='52:54:00:ec:52:44'/> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> ...... </devices> ...... </domain> # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ /sys/bus/pci/drivers/pci-stub # virsh detach-device myVF hostdev.xml error: Failed to detach device from hostdev.xml error: operation failed: network device 52:54:00:a4:8b:17 not found Notes, there are 2 questions in here, the one is libvirt allows to use 'virsh attach-device' to attach a interface XML to guest, but don't allow detach it from guest via 'virsh detach-device'. the other is the error information also is weird, I haven't specified any MAC address in hostdev.xml, it seems 'virsh detach-device' default to detach the first network interface. # virsh detach-interface myVF hostdev 52:54:00:ec:52:44 Interface detached successfully # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver Notes, the network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device. # virsh destroy myVF Domain myVF destroyed # virsh nodedev-reattach pci_0000_03_10_1 Device pci_0000_03_10_1 re-attached # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ /sys/bus/pci/drivers/igbvf Notes, I must manually run 'virsh nodedev-reattach' to explicity return the PCI device to host, I think managed='yes' doesn't make sense in here. Actual results: # readlink -f /sys/bus/pci/devices/0000\:03\:10.1/driver/ /sys/devices/pci0000:00/0000:00:01.0/0000:03:10.1/driver Notes, the network interface with managed='yes' mode, but libvirt hasn't return the host PCI device to host after hot-unplugging PCI device. Expected results: libvirt should return the host PCI device to host with managed='yes' mode after hot-unplugging PCI device. Additional info: