Bug 723535

Description of problem:
Pci-stub device driver disappears after an attempt to boot a guest domain with a PCI device already used by another guest domain. This "breaks" the device for the first virtual guest domain already using it. It is still seen by the guest, but it does not function correctly as dmesg repots issues with the PCI device. Because the link to the pci-stub device driver is lost. The PCI device assigned to the virt guest is a PCIe ethernet card running correctly with enabled VT-d extensions. Assigning to qemu-kvm component for now.

Version-Release number of selected component (if applicable):
libvirt-0.8.7-18.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64
kernel-2.6.32-131.0.15.el6.x86_64

How reproducible:
always

Steps to Reproduce:
1. Configure system for PCI device assignement by enabling VT-d extensions [1]
2. Add PCI device to a virt guest (eth0 at 0000:03:00.0 in this case)
3. Prepare test1 guest domain with attached PCI device 03:00.0
# virsh nodedev-dumpxml pci_0000_03_00_0
<device>
  <name>pci_0000_03_00_0</name>
  <parent>pci_0000_00_01_0</parent>
  <driver>
    <name>tg3</name>
  </driver>
  <capability type='pci'>
    <domain>0</domain>
    <bus>3</bus>
    <slot>0</slot>
    <function>0</function>
    <product id='0x1681'>NetXtreme BCM5761 Gigabit Ethernet PCIe</product>
    <vendor id='0x14e4'>Broadcom Corporation</vendor>
  </capability>
</device>
# virsh start test1
# vmpid=`pgrep qemu-kvm`
# regions=`lspci -vvv -s 03:00.0 | sed -ne 's/Region.*Memory at \([a-f0-9]\{8\}\) .*/-e \1/p'`
# virsh dumpxml test1
... 
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
...

# grep $regions /proc/$vmpid/maps
7f3138138000-7f3138148000 rw-s f7110000 00:00 7138                       /sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0/resource2
7f313836f000-7f313837f000 rw-s f7100000 00:00 7137                       /sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0/resource0
# lspci -vvv -s 03:00.0
...
	Capabilities: [160] Device Serial Number 00-10-18-ff-fe-61-35-9a
	Capabilities: [16c] Power Budgeting <?>
	Kernel driver in use: pci-stub
	Kernel modules: tg3

# readlink /sys/bus/pci/devices/0000\:03\:00.0/driver
../../../../bus/pci/drivers/pci-stub

4. Make sure the device works correctly for the test1 guest.

----
So far we are good. Lets try to start test2 guest domain which will attempt to attach the same PCI device used by test1.
---- 

5. Start test2 guest domain which should fail to attach the same PCI device as we used for test1 (expected to fail to start). And check the driver used for the PCI device again.

# virsh dumpxml test2
... 
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
...
# virsh start test2
error: Failed to start domain test2
error: internal error Not detaching active device 0000:03:00.0

# readlink /sys/bus/pci/devices/0000\:03\:00.0/driver
<nothing here, becasue there is no such file>
# ls /sys/bus/pci/devices/0000\:03\:00.0/driver
ls: cannot access /sys/bus/pci/devices/0000:03:00.0/driver: No such file or directory
# lspci -vvv -s 03:00.0
...
	Capabilities: [16c] Power Budgeting <?>
	Kernel modules: tg3
Good news is, that at least the device is still mapped to memory of qemu-kvm process for test1 domain:
# grep $regions /proc/$vmpid/maps 
7ff1d23bd000-7ff1d23cd000 rw-s f7110000 00:00 7138                       /sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0/resource2
7ff1d37f5000-7ff1d3805000 rw-s f7100000 00:00 7137                       /sys/devices/pci0000:00/0000:00:01.0/0000:03:00.0/resource0

6. Checking whether the device still works for the test1 guest, shows that it does not :(

7. Another attempt to start test2 guest domain (expected to fail)
# virsh start test2
error: Failed to start domain test2
error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/9
Using CPU model "cpu64-rhel6"
Failed to assign device "hostdev0" : Device or resource busy
qemu-kvm: -device pci-assign,host=03:00.0,id=hostdev0,configfd=32,bus=pci.0,addr=0x7: Device 'pci-assign' could not be initialized


Actual results:
The device becomes unusable for the guest using it due to missing driver.

Expected results:
The assigned PCI device should continue working properly for the guest already using it.

Additional info:
Testing this with a live-assignment of the PCI device to test2 is not currently possible due to bug 720972.

Some relevant dmesg output from the host if it helps:
 ...
ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
tg3 0000:03:00.0: PME# enabled
tg3 0000:03:00.0: PCI INT A disabled
pci-stub 0000:03:00.0: claimed by stub
device vnet0 entered promiscuous mode
virbr0: topology change detected, propagating
virbr0: port 2(vnet0) entering forwarding state
pci-stub 0000:03:00.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
pci-stub 0000:03:00.0: restoring config space at offset 0xc (was 0x0, writing 0xf7400000)
pci-stub 0000:03:00.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100002)
assign device: host bdf = 3:0:0
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: Invalid ROM contents
pci-stub 0000:03:00.0: restoring config space at offset 0x1 (was 0x100000, writing 0x100402)
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
vnet0: no IPv6 routers present
kvm: 8524: cpu0 unimplemented perfctr wrmsr: 0xc1 data 0xabcd
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
pci-stub 0000:03:00.0: irq 51 for MSI/MSI-X
tg3 0000:03:00.0: BAR 0: can't reserve mem region [0xf7100000-0xf710ffff]
tg3 0000:03:00.0: Cannot obtain PCI resources, aborting
tg3: probe of 0000:03:00.0 failed with error -16


----
1. http://documentation-stage.bne.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html-single/Virtualization/index.html#chap-Virtualization-PCI_passthrough