Bug 609437

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.

ADDED NOTE:

  My test Guest OS is 'el6.0'

Is this problem specific to SRIOV devices?  Can you reproduce it on an ordinary PCI device?

Yes,it can reproduce on an ordinary PCI device.

test with libvirt-0.8.1-15.el6.
Set 'managed='yes'' in xml, guest won't be shutoff after hot unplug.

When you use managed='no' you really have to reset the device before assigning it to the guest.  There may be some situations in which that's not strictly necessary, but you really have to know the state of the hardware very well.  If you do a nodedev-reset before attaching the device to the guest, does the problem go away?

 Use 'nodedev-reset' to reset the PCI device before attaching to guest,the problem still exists.

Can you confirm that everything works correctly if you use managed="yes" and do not use any of the 'virsh nodedev-detach/reset/reattach' commands.

If using managed=no there are only two support usage scenarios:

Coldplug:

   virsh nodedev-dettach DEVICE
   virsh nodedev-reset DEVICE
   start KVM guest with managed=no
   stop KVM guest
   virsh nodedev-reset DEVICE
   virsh nodedev-reattach DEVICE

Hotplug:

   start KVM guest with no host device
   virsh nodedev-dettach DEVICE
   virsh nodedev-reset DEVICE
   virsh attach-device with managed=no
   virsh detach-device 
   virsh nodedev-reset DEVICE
   virsh nodedev-reattach DEVICE

Hi 'DB',

Steps:
1.# cat pf.xml 
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address bus='0' slot='25' function='0'/>
       </source>
    </hostdev>
2.# virsh attach-device rhel6 pf.xml 
Device attached successfully

3.# virsh detach-device rhel6 pf.xml 
Device detached successfully

After that steps, everything works fine.

wangyimiao confirmed on irc, that only coldplug scenario was working with managed=no, while the problem could still be observed in the hotplug case... not sure what to think about it, I'm trying to reproduce myself

I can't reproduce, it all works for me with managed=no. Even crazy things like starting a guest with PCI device attached and hot-unplugging and plugging back work.

Hi 'JD',
  
Hotplug:

   start KVM guest with no host device
   virsh nodedev-dettach DEVICE
   virsh nodedev-reset DEVICE
   virsh attach-device with managed=no
   virsh detach-device 
   virsh nodedev-reset DEVICE
   virsh nodedev-reattach DEVICE   

That is my PCI device info.

1.# virsh nodedev-dumpxml pci_0000_00_19_0
.......
    <product id='0x10bd'>82566DM-2 Gigabit Network Connection</product>
    <vendor id='0x8086'>Intel Corporation</vendor>
........

 2.After the step "virsh attach-device with managed=no", please login guest to check the PCI device have get the valid IP or it works fine .May be it can help you to reproduce this issue.

The following line was printed by qemu into rhel6.log on the machine where this bug reproduces:

assigned_dev_pci_write_config: pwrite failed, ret = -1 errno = 13

Could you please download and install test packages from http://people.redhat.com/clalance/bz617116/ and check if that makes any difference?

Reproduce the problem with:
libvirt-0.8.1-18clalance.el6.x86_64 
qemu-kvm-0.12.1.2-2.90.el6.x86_64
kernel-2.6.32-53.el6
&
libvirt-0.8.1-18clalance.el6.x86_64 
qemu-kvm-0.12.1.2-2.104.el6.x86_64
kernel-2.6.32-53.el6

Test with : 
libvirt-0.8.1-20.el6.x86_64 
qemu-kvm-0.12.1.2-2.104.el6.x86_64
kernel-2.6.32-53.el6

reproduct it with rhel6 guest, but cannot reproduce it with rhel5 guest.

I talked to Don Dutile a bit about this bug, and we now have a theory as to why this works with managed=yes but not managed=no.

The first thing to realize is that the 82576 is an SR-IOV capable device.  As such, it is required by the SR-IOV spec to support FLR (function-level reset).  For historical reasons, when libvirt detects that a device supports FLR, it does *absolutely nothing* when a reset is requested.  In the case for starting or stopping a virtual machine, this is actually the correct behavior; qemu/kvm will do the FLR on our behalf, and all will be right with the world.  This is why "managed=yes" works just fine.

However, when you do a "virsh nodedev-reset", there is no qemu active to do the reset for us.  So in this case, we will come into the same code path, determine that the device supports FLR, and still do nothing.  But in this case, there is no qemu in the mix to actually do the reset on our behalf, so nothing at all happens.  Now we start the guest, but nothing has ever reset the device, so things get confused.

We may be able to confirm this theory by playing around with /sys/bus/pci/devices/<pciid>/reset; writing a "1" into this file should cause a FLR to occur (though only for SR-IOV VF devices).  After the nodedev-reset step in the above reproducers, try to echo 1 > /sys/bus/pci/devices/<pciid>/reset, and then follow the rest of the steps to see if you can still reproduce the problem.

Chris Lalancette

Alex Williamson has made a couple of bug fixes in qemu-kvm &/or kvm support when hot-unplug done.
Please update to latest qemu-kvm(we're up to something like .100+) & kernel(like .55+) and retest.

I test it with the latest qemu-kvm-0.12.1.2-2.108.el6.x86_64 and kernel 2.6.32-59.el6.x86_64 and the error still exists.

libvirt-0.8.1-21.el6.x86_64

Finally I found out the core of this bug. It's not related to any particular network card. The Chinese boxes are just lucky enough to hit the bug, unlike other boxes I tried to test this on.

The problem is this:

With svirt enabled, libvirt relabels /sys files corresponding to given PCI device so that qemu-kvm can access them and then it instructs qemu-kvm to attach the device. On detach, libvirt issues 'device_del' qemu command and after it returns, it relabels the files back, which prevents qemu-kvm from accessing them.

Unfortunately, device_del is not synchronous for PCI devices, it merely asks the guest to release the device and returns. If the guest doesn't release it quickly enough, libvirt relabels the files before the guest/qemu-kvm is finished with the PCI device, which may result in the EACCES failure mentioned in comment 14. qemu-kvm's reaction on such failure is exit() and the guest disappears.

There's no way we could fix this properly for 6.0. We can only hack around it...

I think qemudReattachManagedDevice() might be the reason why we didn't see this bug in managed=yes case. The function is called after device_del returns but before any relabeling is done. The function looks like this:

    int retries = 100;

    if (pciDeviceGetManaged(dev)) {
        while (pciWaitForDeviceCleanup(dev, "kvm_assigned_device")
               && retries) {
            usleep(100*1000);
            retries--;
        }
        if (pciReAttachDevice(dev) < 0) {
            virErrorPtr err = virGetLastError();
            VIR_ERROR(_("Failed to re-attach PCI device: %s"),
                      err ? err->message : "");
            virResetError(err);
        }
    }

In managed=yes case we wait up to 10 seconds in the first loop until the device is actually freed by kvm. Should we do that for non-managed devices as well?

The bad thing is I wasn't able to prove that this code is what helps. If I commented the call to this function in a hope to reproduce the bug with managed=yes, it didn't reproduce even for managed=no.

Hmm, actually, after rebooting the host, I reproduced the bug for managed=yes with the call to qemudReattachManagedDevice() commented out...

libvirt-0_8_1-22_el6 has been built in RHEL-6-candidate with the fix.

Dave

Verified PASSED with 
libvirt-client-0.8.1-22.el6.x86_64
qemu-kvm-0.12.1.2-2.108.el6.x86_64
qemu-img-0.12.1.2-2.108.el6.x86_64
.

Red Hat Enterprise Linux 6.0 is now available and should resolve
the problem described in this bug report. This report is therefore being closed
with a resolution of CURRENTRELEASE. You may reopen this bug report if the
solution does not work for you.