Bug 521346 - PCI-Paththrough with PCI-Card does not work anymore with RHEL5.4
Summary: PCI-Paththrough with PCI-Card does not work anymore with RHEL5.4
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen
Version: 5.4
Hardware: All
OS: Linux
urgent
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks: 525149
TreeView+ depends on / blocked
 
Reported: 2009-09-04 22:32 UTC by Jens Kuehnel
Modified: 2010-04-08 16:25 UTC (History)
9 users (show)

Fixed In Version: xen-3.0.3-95.el5
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-03-30 08:57:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
The correct logfile (13.83 KB, text/plain)
2009-09-07 10:24 UTC, Jens Kuehnel
no flags Details
Ignore unimplemented PHYSDEVOP_map_pirq (1.87 KB, patch)
2009-09-18 09:23 UTC, Jiri Denemark
no flags Details | Diff


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2010:0294 0 normal SHIPPED_LIVE xen bug fix and enhancement update 2010-03-29 14:20:32 UTC

Description Jens Kuehnel 2009-09-04 22:32:14 UTC
Description of problem:
Starting a domU (PVM) with an attached PCI-Card creates an errormessage.
Worked with RHEL 5.3

Version-Release number of selected component (if applicable):
xen-3.0.3-94.el5

How reproducible:
100%

Steps to Reproduce:
1. configure pciback
2. add (pci-dev-assign-strict-check no) to /etc/xen/xend-config.sxp
3. reboot
4. create domU 
5. add PCI-Card to vm like pci = [ '01:00.0']
6. xm create domU
  
Actual results:
Error: (38, 'Function not implemented')


Expected results:
DomU starts

Additional info:

Comment 1 Jiri Denemark 2009-09-07 07:42:21 UTC
Hmm, everything works as expected for me. Could you please provide xend.log and output of lspci?

Thanks.

Comment 2 Jens Kuehnel 2009-09-07 08:08:13 UTC
by the way it works with xen-3.0.3-80.el5_3.3.x86_64.

here the important information:

Part of modprobe.conf:
options pciback hide=(01:01.0)(01:01.1)(01:00.0)

Part of lspci:
01:00.0 Network controller: AVM GmbH A1 ISDN [Fritz] (rev 02)
01:01.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
01:01.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
01:02.0 SCSI storage controller: Adaptec AIC-7892A U160/m (rev 02)
01:03.0 VGA compatible controller: XGI Technology Inc. (eXtreme Graphics Innovation) Z7/Z9 (XG20 core)

Part of /etc/xen/SAM:
pci = [ '01:00.0']

Part of /etc/xen/xend-config.sxp
(pci-dev-assign-strict-check no)

Part of xend.log:
[2009-09-04 23:34:59 xend 4922] DEBUG (DevController:116) DevController: writing
 {'domain': 'SAM', 'frontend': '/local/domain/1/device/vbd/51761', 'format': 'ra
w', 'dev': 'xvdd1', 'state': '1', 'params': '/dev/gaias/SAMdata', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'phy'} to /local/domain/0/backend/vbd/1/51761.xvdd1: [Errno 2] No such file or directory: '/dev/xvdd1'
[2009-09-04 23:34:59 xend 4922] INFO (pciquirk:91) NO quirks found for PCI device [1244:0a00:1244:0a00]761', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'b[2009-09-04 23:34:59 xend 4922] DEBUG (pciquirk:131) Permissive mode NOT enabled for PCI device [1244:0a00:1244:0a00]
[2009-09-04 23:34:59 xend 4922] DEBUG (pciif:357) pci: enabling ioport 0xa480/0x20'domain': 'SAM', 'frontend': '/local/domain/1/device/vbd/51761', 'format': 'ra[2009-09-04 23:34:59 xend 4922] DEBUG (pciif:370) pci: enabling iomem 0xfbbbac00/0x20 pfn 0xfbbba/0x1
[2009-09-04 23:34:59 xend.XendDomainInfo 4922] ERROR (XendDomainInfo:218) Domain construction failed4:0a00:1244:0a00]
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 211, in createon failed
    vm.initDomain()
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 1975, in initDomainn()
    raise VmError(str(exn))
VmError: (38, 'Function not implemented')
[2009-09-04 23:34:59 xend.XendDomainInfo 4922] DEBUG (XendDomainInfo:2119) XendDomainInfo.destroy: domid=1

Comment 4 Jiri Denemark 2009-09-07 08:38:20 UTC
Thanks for the additional info. Do you run kernel from 5.3, i.e. 2.6.18-128.*?

BTW, please, provide logs as attachments instead of pasting them into a comment next time to avoid corruptions.

Comment 5 Jens Kuehnel 2009-09-07 10:24:49 UTC
Created attachment 359979 [details]
The correct logfile

Comment 6 Jens Kuehnel 2009-09-07 10:32:25 UTC
I did an update th RHEL5.4. Because of Bug 521345 I had to boot with kernel-xen-2.6.18-128.7.1.el5.x86_64. Because of this bug 521346 I had to revert to xen-3.0.3-80.el5_3.3.x86_64.

The attached logfile contains two trys. At 23:00:19 I tried to boot with pcipermissive disabled, at 20:00:56 I tried it with pcipermissive enabled.

Do you use PCI oder PCIe cards for that? What Caps does you PCI card have?
Because my AVM has none.

Thanks

Comment 7 Jiri Denemark 2009-09-18 09:23:42 UTC
Created attachment 361615 [details]
Ignore unimplemented PHYSDEVOP_map_pirq

Comment 10 Jens Kuehnel 2009-09-18 17:50:49 UTC
Yes, the fix in #7 works perfectly. 
Can you please insert that into the next xen.rpm? 

Thanks.
Greeting from Frankfurt,Germany.
Jens Kühnel

Comment 11 Jiri Denemark 2009-09-19 09:30:07 UTC
Thanks for the testing. Of course, this fix will be included in the next package.

Comment 16 Jiri Denemark 2009-09-22 09:32:32 UTC
Fix built into xen-3.0.3-95.el5

Comment 18 Yewei Shao 2009-09-28 05:25:27 UTC
I try to reproduce this bug by the steps that describing in this bug, but for the step 1, need to configure pciback fist. For this step, I use to add "pciback.permissive pciback.hide=(00:19.0)' in the grub file in kernel line, and when try to reboot the system, then I will get the following error message that "Unknow boot option 'pciback.permissive' " and the error message of "Unknow boot option 'pciback hide=(00:19.0)' ". So for the step 1, how to do it? If my steps is right? If it is right, how to solve the error message? Thanks.

part of lspci:
[root@localhost ~]# lspci
00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port (rev 03)
00:02.0 VGA compatible controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller (rev 03)
00:02.1 Display controller: Intel Corporation 4 Series Chipset Integrated Graphics Controller (rev 03)
00:03.0 Communication controller: Intel Corporation 4 Series Chipset HECI Controller (rev 03)
00:03.2 IDE interface: Intel Corporation 4 Series Chipset PT IDER Controller (rev 03)
00:03.3 Serial controller: Intel Corporation 4 Series Chipset Serial KT Controller (rev 03)
00:19.0 Ethernet controller: Intel Corporation 82567LM-3 Gigabit Network Connection (rev 02)

Comment 19 Jens Kuehnel 2009-09-28 05:35:08 UTC
pciback is a modul try adding 

options pciback hide=(01:01.0)(01:01.1)(01:00.0)(01:02.0)

to /etc/modprobe.conf.

That works for me.

Comment 20 Yewei Shao 2009-09-28 05:57:07 UTC
Can you tell me if I add the "options pciback hide=(00:19.0)" to the /etc/modprobe.conf file, should I still need to add the "pciback.permissive pciback.hide=(00:19.0)" to the grub file? If it should, still get the same error message like the comment #18. Thanks.

Comment 21 Jens Kuehnel 2009-09-28 06:12:25 UTC
no, the kernel option is only for build in modules.

Comment 22 Chris Lalancette 2009-09-28 07:37:30 UTC
Right, exactly.  In RHEL, pciback is a module, so you have to use module options, not kernel command-line options.

Note that you can do this all "live", as well, meaning that you don't have to use module options.  Here's the script I use:

#!/bin/bash

modprobe pciback
echo -n $1 > /sys/bus/pci/drivers/forcedeth/unbind
echo -n $1 > /sys/bus/pci/drivers/pciback/new_slot
echo -n $1 > /sys/bus/pci/drivers/pciback/bind

You then run this script with something like:

# pcihide 0000:00:09.0

You find out the number that you are using using lspci, and then prepend 0000 to the front of it.  Basically what the script does is to load the pciback module, unbind the device from the old driver, then bind it to pciback so nothing else can grab the device.

Chris Lalancette

Comment 23 Yewei Shao 2009-09-28 09:02:43 UTC
When I try to do this bug in the ia64 system, and first try to dettach pci from the host, but it will failed.
I can use "virsh nodedev-dumpxml $pci" to see the pci device, but then I try to use "virsh nodedev-dettach $pci", then it failed and report error message like: " failed to load pci-stub or pci drivers. Internal error cannot find any PCI stub module". 
This error can only happens in my ia64 system, and works fine in my i386 and x86_64 system.

Comment 24 Chris Lalancette 2009-09-28 09:21:46 UTC
(In reply to comment #23)
> When I try to do this bug in the ia64 system, and first try to dettach pci from
> the host, but it will failed.
> I can use "virsh nodedev-dumpxml $pci" to see the pci device, but then I try to
> use "virsh nodedev-dettach $pci", then it failed and report error message like:
> " failed to load pci-stub or pci drivers. Internal error cannot find any PCI
> stub module". 
> This error can only happens in my ia64 system, and works fine in my i386 and
> x86_64 system.  

Yes, as far as I know we don't support pci passthrough on ia64.

Chris Lalancette

Comment 27 Yewei Shao 2009-12-31 07:02:21 UTC
I verify this bug by following steps:
1. configure pciback by following:
   (1) load pciback module via modprobe, in domain0:
      # modprobe pciback
      # lsmod | grep pciback 
   (2) Check the native driver with which the pci device is originally bond to.
     # ll /sys/bus/pci/devices/{pci-idenfier}/driver
   (3) Bind pci device to pciback driver
     # echo -n {pci_idenfier} > /sys/bus/pci/drivers/{native_dirver}/unbind
     # echo -n {pci_idenfier} > /sys/bus/pci/drivers/pciback/new_slot
     # echo -n {pci_idenfier} > /sys/bus/pci/drivers/pciback/bind
   (4) Check that pci device has been hidden from domain0 successfully
     # xm pci-list-assignable-devices

2. add (pci-dev-assign-strict-check no) to /etc/xen/xend-config.sxp
3. restart the xend
4. create domU 
5. add PCI-Card to vm like pci = [ '01:00.0']
6. xm create domU
7. In domain0, check pci device that assigned to the guest:
   # xm pci-list $domain_id
8. In the PV guest, check if the pci device has been assigned to it successfully:
   # lspci

The domain can be start up successfully with attached PCI-Card. So this bug is verified in xen-3.0.3-102.el5.

Comment 29 errata-xmlrpc 2010-03-30 08:57:36 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0294.html

Comment 30 Paolo Bonzini 2010-04-08 15:43:19 UTC
This bug was closed during 5.5 development and it's being removed from the internal tracking bugs (which are now for 5.6).


Note You need to log in before you can comment on or make changes to this bug.