Bug 706869

Summary: SR-IOV does not work, allegedly due to lack of ACS
Product: Red Hat Enterprise Linux 6 Reporter: Troels Arvin <troels>
Component: libvirtAssignee: Osier Yang <jyang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.1CC: ajia, charles_rose, dallan, dyuan, gren, linux-bugs, matt_domsch, mzhan, rwu, surya_prabhakar, veillard, zpeng
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-0.9.2-1.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 11:09:29 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Output from "lspci" while SR-IOV is enabled
none
Output from "lspci -vvv"
none
output from "lspci -vvv" : verify machine none

Description Troels Arvin 2011-05-23 10:01:32 UTC
Created attachment 500385 [details]
Output from "lspci" while SR-IOV is enabled

Description of problem:
I followed the instructions at http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization/sect-Para-virtualized_Windows_Drivers_Guide-How_SR_IOV_Libvirt_Works.html and added two virtual-function PCIe network ports to a virtual server, the virtual server will not start. The following message is seen in syslog:

libvirtd: 11:20:08.885: 4886: error : pciDeviceIsAssignable:1487 : this function is not supported by the connection driver: Device 0000:07:10.0 is behind a switch lacking ACS and cannot be assigned

With such new hardware, shouldn't SR-IOV be working?

How reproducible:
Every time.

Steps to Reproduce:
1. Grab a new Dell Poweredge R710 server with an Intel 82576 Gigabit ET Quad Port Server Adapter and apply all firmware updates
2. Make sure that SR-IOV is enabled in bios.
3. Add to virtual-function PCI devices to a virtual host, using virt-manager.
  
Actual results:
Virtual server doesn't start.

Expected results:
Virtual server starts.

Additional info:

Comment 2 Troels Arvin 2011-05-23 10:08:12 UTC
Created attachment 500386 [details]
Output from "lspci -vvv"

Comment 3 Dave Allan 2011-05-23 11:18:00 UTC
Regardless of whether the reporter's hardware supports ACS, I think the error is wrong:

this function is not supported by the connection driver

seems incorrect; it's supported by the connection driver, but not the hardware.

Comment 4 Troels Arvin 2011-05-23 14:22:43 UTC
It sounds like the Dell R710 simply doesn't support PCIe Access Control Services (ACS):
http://thread.gmane.org/gmane.linux.hardware.dell.poweredge/40981

So I guess this bug can be closed.


By the way: The R710 uses the Intel 5520 chipset. A bit of googling reveals a hard to decode note about (lack of?) ACS support: http://www.intel.com/assets/pdf/specupdate/321329.pdf

It would be great if Red Hat published a list of server hardware combinations which is verified to support SR-IOV including ACS.

Comment 5 Dave Allan 2011-05-24 15:15:25 UTC
Hi Troels, thanks for the additional info.  I'm going to leave the BZ open to address my comment 3.

Comment 6 Osier Yang 2011-05-31 07:25:35 UTC
patch posted to upstream, http://www.redhat.com/archives/libvir-list/2011-May/msg01867.html

Comment 7 Osier Yang 2011-06-02 07:52:51 UTC
patch got pushed in upstream, move to POST.

Comment 8 Daniel Veillard 2011-06-23 03:18:31 UTC
For the record this is upstream commit a74575e374e850999d624e192a9fbb30829085a1

Daniel

Comment 9 Daniel Veillard 2011-06-23 03:18:50 UTC
This should be fixed by the libvirt-0.9.2-1.el6 rebase

Comment 13 Troels Arvin 2011-07-01 08:11:24 UTC
Unfortunately, it may take quite a while before I can get a service window where I can test the updated libvirt -- "quite a while" meaning up to ½ a year :-(

Comment 15 dyuan 2011-07-05 11:34:56 UTC
Hi, Dallan

We have a 4 port 82576 with non-ACS switch which is used to verify the bug 526713, fail to start the guest(assign the PF to the guest) with the same error in the bug description "this function is not supported by the connection driver".
But this card does not supported VF creation(the reporter used the VF in step 3 in comment 0), need to confirm with you whether we can use this card on this bug verification ?

If not, we'll try to get a new device with non-ACS switch and support VF asap, thanks.

YuanDan

Comment 16 Dave Allan 2011-07-05 18:33:50 UTC
dyuan, you have to talk to the bug assignee about this.

Comment 17 Osier Yang 2011-07-06 03:17:46 UTC
Hi, dyuan,

Could you attach the debug log, I don't think it relates with whether it's PF and VF, testing both of them shouldn't see the error (this function is not supported by driver) again. If you can attach the debug log, it will be clear which function throws the error, and so I can see if my patch missed fix on some function or if the patch is not applied.

Comment 18 zhe peng 2011-07-06 06:12:11 UTC
Created attachment 511443 [details]
output from "lspci -vvv" : verify machine

verified on:
libvirt-0.9.2-1.el6
kernel-2.6.32-156.el6
qemu-kvm-0.12.1.2-2.165.el6

#virsh start demo
error: Failed to start domain demo
error: internal error Device 0000:05:00.0 is behind a switch lacking ACS and can not be assigned
#tail -f /var/log/libvirt/libvirtd.log
......
20:55:48.466: 15475: error : pciDeviceIsAssignable:1649 : internal error Device 0000:05:00.0 is behind a switch lacking ACS and cannot be assigned
......

Comment 20 Rita Wu 2011-07-06 10:20:47 UTC
Set it as VERIFIED per comment18

Comment 21 errata-xmlrpc 2011-12-06 11:09:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html