Bug 499678

Summary: libvirt should be able to reset a PCI function even if it causes other unused devices/functions to be reset
Product: [Fedora] Fedora Reporter: Charles Rose <charles_rose>
Component: libvirtAssignee: Mark McLoughlin <markmc>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 11CC: berrange, clalance, crobinso, itamar, markmc, paniraja_km, veillard, virt-maint, wwlinuxengineering
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: 0.6.2-17.fc11 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-09-04 00:09:45 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 480594    
Attachments:
Description Flags
lspci -tv
none
lspci -vvv
none
Syslog with virsh nodedev-reset
none
Syslog with virt-manager PCI Assignment
none
lspci -t -v output
none
lspci -vvv output none

Description Charles Rose 2009-05-07 12:02:29 EDT
Created attachment 342865 [details]
lspci -tv

While testing virt-manager's device assignment with the following steps:

  * Run virt-manager, open an existing guest and go to the details tab
  * Click "Add hardware", choose "Physical host device" and click "Forward"
  * Choose the appropriate device from the drop down list (This is the second LOM):
     "01:00.1 NetXtreme II BCM5716 Gigabit Ethernet"
  * Click "Forward" and "Finish"

and it failed with:

Uncaught error adding device: Could not detach PCI device: this function is not supported by the hypervisor: No PCI reset capability available for 0000:01:00.1

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/addhardware.py", line 571, in finish
    self.add_hostdev()
  File "/usr/share/virt-manager/virtManager/addhardware.py", line 607, in add_hostdev
    self._dev.setup()
  File "/usr/lib/python2.6/site-packages/virtinst/VirtualHostDevice.py", line 208, in setup
    raise RuntimeError(_("Could not detach PCI device: %s" % str(e)))
RuntimeError: Could not detach PCI device: this function is not supported by the hypervisor: No PCI reset capability available for 0000:01:00.1

I will test this with another network card.
Comment 1 Charles Rose 2009-05-07 12:03:27 EDT
I am using python-virtinst-0.400.3-8.fc11
Comment 2 Mark McLoughlin 2009-05-07 12:36:33 EDT
charles: could you attach the output of "lspci -vvv" and "lspi -t -v" ? Are there any libvirtd messages output to /var/log/messages at the time?

it looks like this is one function of a multi-function PCI device, and you're hitting this case:

/* Secondary Bus Reset is our sledgehammer - it resets all
 * devices behind a bus.
 */
static int
pciTrySecondaryBusReset(virConnectPtr conn, pciDevice *dev)
{
...
    /* For now, we just refuse to do a secondary bus reset                   
     * if there are other devices/functions behind the bus.
     * In future, we could allow it so long as those devices
     * are not in use by the host or other guests.
     */
    if (pciBusContainsOtherDevices(conn, dev)) {
	VIR_WARN("Other devices on bus with %s, not doing bus reset",
                 dev->name);
        return -1;
    }

or this case:

/* Power management reset attempts to reset a device using a
 * D-state transition from D3hot to D0. Note, in detect_pm_reset()
 * above we require the device supports a full internal reset.
 */
static int
pciTryPowerManagementReset(virConnectPtr conn, pciDevice *dev)
{
...
    /* For now, we just refuse to do a power management reset
     * if there are other functions on this device.
     * In future, we could allow it so long as those functions
     * are not in use by the host or other guests.
     */
    if (pciDeviceContainsOtherFunctions(conn, dev)) {
	VIR_WARN("%s contains other functions, not resetting", dev->name);
        return -1;
    }

We need to improve this code so that it will reset reset a function, even if the reset affects other functions/devices, so long as the device has been detached in the host and is not in use by any other guest
Comment 3 Mark McLoughlin 2009-05-07 12:45:56 EDT
clarifying ...

before we can reset:

  1) a function on a multi-function device, using PM reset
  2) a device sharing a bus with other devices, using a bus reset

we must check that the other affected functions/devices:

  a) are detached in the host - i.e. bound to pci-stub
  b) not in use by any other guest
  c) not in use by this guest, in the case of hotplug
Comment 4 Charles Rose 2009-05-07 12:50:42 EDT
Created attachment 342879 [details]
lspci -vvv
Comment 5 Charles Rose 2009-05-07 13:09:17 EDT
Created attachment 342887 [details]
Syslog with virsh nodedev-reset
Comment 6 Charles Rose 2009-05-07 13:10:57 EDT
Created attachment 342889 [details]
Syslog with virt-manager PCI Assignment
Comment 7 Mark McLoughlin 2009-05-07 13:17:19 EDT
Right, there we go - libvirt wants to do a Secondary Bus Reset, but it can't because it would also reset the other port on the broadcom NIC:

  warning : Other devices on bus with 0000:01:00.1, not doing bus reset

By the way, even if we fix libvirt as I describe above, you will not be able to use one port in the guest while the other port is in use by the host or another guest
Comment 8 Daniel Berrange 2009-05-07 13:26:34 EDT
Urgh, we really ought to get this kind of message back to the application

The error reported is:

"Could not detach PCI device: this function is not supported by
the hypervisor: No PCI reset capability available for 0000:01:00.1"

It should really have said something more like

"Could not detach PCI device: Other in-use devices on the same bus as device 0000:01:00.1, and FLR is not available"
Comment 9 Mark McLoughlin 2009-05-07 13:41:25 EDT
(In reply to comment #8)

> It should really have said something more like
> 
> "Could not detach PCI device: Other in-use devices on the same bus as device
> 0000:01:00.1, and FLR is not available"  

Agreed it should be better, all the possible reasons are:

  1) device doesn't implement FLR
  2) can't use bus reset because it affects other devices
  3) device doesn't implement PM reset
  4) can't use PM reset on multi-function device
  5) something went wrong doing FLR, SBR or PMR
Comment 10 Paniraj 2009-05-08 02:18:23 EDT
Additinal info to the issue:

I updated the packages again today. Added a Intel dual port NIC. Tried adding this add-on NIC to a guest. I got the same error as before.


Tried, resetting the NIC from virtual shell, I got the same error. (No PCI reset capability available for 0000:84:00.0)

Am attaching lspci outputs of my config.
Comment 11 Paniraj 2009-05-08 02:20:11 EDT
Created attachment 343051 [details]
lspci -t -v output
Comment 12 Paniraj 2009-05-08 02:21:10 EDT
Created attachment 343052 [details]
lspci -vvv output
Comment 13 Noura El hawary 2009-06-09 17:48:22 EDT
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Comment 15 Fedora Update System 2009-08-19 12:34:52 EDT
libvirt-0.6.2-16.fc11 has been submitted as an update for Fedora 11.
http://admin.fedoraproject.org/updates/libvirt-0.6.2-16.fc11
Comment 16 Fedora Update System 2009-08-20 16:58:11 EDT
libvirt-0.6.2-17.fc11 has been pushed to the Fedora 11 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update libvirt'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F11/FEDORA-2009-8790
Comment 17 Fedora Update System 2009-09-04 00:09:12 EDT
libvirt-0.6.2-17.fc11 has been pushed to the Fedora 11 stable repository.  If problems still persist, please make note of it in this bug report.