Bug 706869 - SR-IOV does not work, allegedly due to lack of ACS
Summary: SR-IOV does not work, allegedly due to lack of ACS
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Osier Yang
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-05-23 10:01 UTC by Troels Arvin
Modified: 2011-12-06 11:09 UTC (History)
12 users (show)

Fixed In Version: libvirt-0.9.2-1.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-06 11:09:29 UTC
Target Upstream Version:


Attachments (Terms of Use)
Output from "lspci" while SR-IOV is enabled (6.09 KB, text/plain)
2011-05-23 10:01 UTC, Troels Arvin
no flags Details
Output from "lspci -vvv" (161.13 KB, text/plain)
2011-05-23 10:08 UTC, Troels Arvin
no flags Details
output from "lspci -vvv" : verify machine (68.85 KB, text/plain)
2011-07-06 06:12 UTC, zhe peng
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:1513 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2011-12-06 01:23:30 UTC

Description Troels Arvin 2011-05-23 10:01:32 UTC
Created attachment 500385 [details]
Output from "lspci" while SR-IOV is enabled

Description of problem:
I followed the instructions at http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization/sect-Para-virtualized_Windows_Drivers_Guide-How_SR_IOV_Libvirt_Works.html and added two virtual-function PCIe network ports to a virtual server, the virtual server will not start. The following message is seen in syslog:

libvirtd: 11:20:08.885: 4886: error : pciDeviceIsAssignable:1487 : this function is not supported by the connection driver: Device 0000:07:10.0 is behind a switch lacking ACS and cannot be assigned

With such new hardware, shouldn't SR-IOV be working?

How reproducible:
Every time.

Steps to Reproduce:
1. Grab a new Dell Poweredge R710 server with an Intel 82576 Gigabit ET Quad Port Server Adapter and apply all firmware updates
2. Make sure that SR-IOV is enabled in bios.
3. Add to virtual-function PCI devices to a virtual host, using virt-manager.
  
Actual results:
Virtual server doesn't start.

Expected results:
Virtual server starts.

Additional info:

Comment 2 Troels Arvin 2011-05-23 10:08:12 UTC
Created attachment 500386 [details]
Output from "lspci -vvv"

Comment 3 Dave Allan 2011-05-23 11:18:00 UTC
Regardless of whether the reporter's hardware supports ACS, I think the error is wrong:

this function is not supported by the connection driver

seems incorrect; it's supported by the connection driver, but not the hardware.

Comment 4 Troels Arvin 2011-05-23 14:22:43 UTC
It sounds like the Dell R710 simply doesn't support PCIe Access Control Services (ACS):
http://thread.gmane.org/gmane.linux.hardware.dell.poweredge/40981

So I guess this bug can be closed.


By the way: The R710 uses the Intel 5520 chipset. A bit of googling reveals a hard to decode note about (lack of?) ACS support: http://www.intel.com/assets/pdf/specupdate/321329.pdf

It would be great if Red Hat published a list of server hardware combinations which is verified to support SR-IOV including ACS.

Comment 5 Dave Allan 2011-05-24 15:15:25 UTC
Hi Troels, thanks for the additional info.  I'm going to leave the BZ open to address my comment 3.

Comment 6 Osier Yang 2011-05-31 07:25:35 UTC
patch posted to upstream, http://www.redhat.com/archives/libvir-list/2011-May/msg01867.html

Comment 7 Osier Yang 2011-06-02 07:52:51 UTC
patch got pushed in upstream, move to POST.

Comment 8 Daniel Veillard 2011-06-23 03:18:31 UTC
For the record this is upstream commit a74575e374e850999d624e192a9fbb30829085a1

Daniel

Comment 9 Daniel Veillard 2011-06-23 03:18:50 UTC
This should be fixed by the libvirt-0.9.2-1.el6 rebase

Comment 13 Troels Arvin 2011-07-01 08:11:24 UTC
Unfortunately, it may take quite a while before I can get a service window where I can test the updated libvirt -- "quite a while" meaning up to ½ a year :-(

Comment 15 dyuan 2011-07-05 11:34:56 UTC
Hi, Dallan

We have a 4 port 82576 with non-ACS switch which is used to verify the bug 526713, fail to start the guest(assign the PF to the guest) with the same error in the bug description "this function is not supported by the connection driver".
But this card does not supported VF creation(the reporter used the VF in step 3 in comment 0), need to confirm with you whether we can use this card on this bug verification ?

If not, we'll try to get a new device with non-ACS switch and support VF asap, thanks.

YuanDan

Comment 16 Dave Allan 2011-07-05 18:33:50 UTC
dyuan, you have to talk to the bug assignee about this.

Comment 17 Osier Yang 2011-07-06 03:17:46 UTC
Hi, dyuan,

Could you attach the debug log, I don't think it relates with whether it's PF and VF, testing both of them shouldn't see the error (this function is not supported by driver) again. If you can attach the debug log, it will be clear which function throws the error, and so I can see if my patch missed fix on some function or if the patch is not applied.

Comment 18 zhe peng 2011-07-06 06:12:11 UTC
Created attachment 511443 [details]
output from "lspci -vvv" : verify machine

verified on:
libvirt-0.9.2-1.el6
kernel-2.6.32-156.el6
qemu-kvm-0.12.1.2-2.165.el6

#virsh start demo
error: Failed to start domain demo
error: internal error Device 0000:05:00.0 is behind a switch lacking ACS and can not be assigned
#tail -f /var/log/libvirt/libvirtd.log
......
20:55:48.466: 15475: error : pciDeviceIsAssignable:1649 : internal error Device 0000:05:00.0 is behind a switch lacking ACS and cannot be assigned
......

Comment 20 Rita Wu 2011-07-06 10:20:47 UTC
Set it as VERIFIED per comment18

Comment 21 errata-xmlrpc 2011-12-06 11:09:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html


Note You need to log in before you can comment on or make changes to this bug.