Bug 738987 - kernel panic when reset 82574L via pci-sysfs
Summary: kernel panic when reset 82574L via pci-sysfs
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel
Version: 6.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Red Hat Kernel Manager
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
Depends On:
Blocks: 767187
TreeView+ depends on / blocked
 
Reported: 2011-09-16 09:17 UTC by Chao Yang
Modified: 2013-01-10 08:22 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-12-16 21:14:30 UTC


Attachments (Terms of Use)
guest device message (21.99 KB, text/plain)
2011-09-16 09:20 UTC, Chao Yang
no flags Details
host dmesg (72.04 KB, text/plain)
2011-09-16 09:21 UTC, Chao Yang
no flags Details

Comment 1 Chao Yang 2011-09-16 09:20:41 UTC
Created attachment 523522 [details]
guest device message

Comment 2 Chao Yang 2011-09-16 09:21:36 UTC
Created attachment 523523 [details]
host dmesg

Comment 3 Chao Yang 2011-09-16 09:22:50 UTC
not sure whether it is the correct component, if not, please help change to the correct one. Thanks!

Comment 4 Chao Yang 2011-09-16 09:24:25 UTC
# cat /proc/cmdline 
ro root=/dev/mapper/vg_amd616881-lv_root rd_NO_LUKS rd_LVM_LV=vg_amd616881/lv_swap LANG=en_US.UTF-8 rd_LVM_LV=vg_amd616881/lv_root rd_NO_MD  SYSFONT=latarcyrheb-sun16 rhgb crashkernel=128M  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM amd_iommu=on console=tty0 console=ttyS0,115200

Comment 6 Alex Williamson 2011-09-20 18:57:02 UTC
Can this be reproduced using the PCI sysfs reset file directly?  For example, bind the device to pci-stub, then do the following without assigning the device to a guest:

# echo 1 > /sys/bus/pci/devices/0000:06:00.0/reset

If that also fails (expected) then this is probably more appropriately a kernel issue and we'll have to create a reset quirk for this device.

Comment 7 Chao Yang 2011-09-22 08:07:34 UTC
(In reply to comment #6)
> Can this be reproduced using the PCI sysfs reset file directly?  For example,
> bind the device to pci-stub, then do the following without assigning the device
> to a guest:
> 
> # echo 1 > /sys/bus/pci/devices/0000:06:00.0/reset
> 
> If that also fails (expected) then this is probably more appropriately a kernel
> issue and we'll have to create a reset quirk for this device.

Bind the device to pci-stub, and then echo to PCI sysfs reset file directly, got following error:
# echo 1 > /sys/bus/pci/devices/0000\:06\:00.0/reset 
-bash: echo: write error: Invalid argument

If boot a guest with it assigned then shut it down, successful to echo to PCI sysfs reset file, but *won't* trigger kernel panic

Comment 9 Alex Williamson 2011-09-22 16:59:03 UTC
I can trigger this without KVM being involved.  The mistake I made in the Comment 6 instructions was to bind the device to pci-stub.  In fact, we should leave the device bound to e1000e, then we just need to echo 1 to the pci-sysfs reset file (sometimes more than once) and the system will hit the same panic.  I'd like to see if it's avoided if we boot with noaer, but I seem to have lost the test system.

Comment 14 Alex Williamson 2011-09-26 17:22:57 UTC
pci=noaer makes no difference since this is an APEI error. ghes.disable=1 will avoid printing the error at the OS and not call panic, but we still get an unknown NMI and the device doesn't work after.

Based on the errata for this 82574L, I also tried masking unsupported request errors in the advanced error reporting capability register (config offset 0x108, bit 20).  This does seem to prevent the APEI error, but the device is unable to dhcp an IP after reset.  This possibly relates back to Specification Clarification note 2 in the specification update for this device, which indicates a D3->D0 transition resets the PHY.

I also attempted to do a secondary bus reset on the parent bridge to this device from userspace, but met another APEI error for an unsupported request from the bridge.

Adding Matthew Garrett and Don Zickus to the cc since they may be able to shed some light on the ACPI and APEI aspects of this.

I did verify this system is running the latest released BIOS.

Comment 15 Matthew Garrett 2011-09-26 17:35:29 UTC
If it's generating an NMI even without any APEI/AER support, I think it's pretty clear that the hardware is doing something nasty to the bus and we're just the messenger.

Comment 16 Don Zickus 2011-09-26 20:00:33 UTC
(In reply to comment #15)
> If it's generating an NMI even without any APEI/AER support, I think it's
> pretty clear that the hardware is doing something nasty to the bus and we're
> just the messenger.

Right or the hardware's firmware just croaked.  Either way an NMI is being generated.  In the APEI case, the bios firmware traps it, grabs some info and wraps a pretty little acpi bow on it.  Without APEI, the NMI just goes to the OS to decipher, which we do not do a good job of currently.

There isn't much we can do except ask for a firmware update on the hardware or just blacklist the device from being used with a guest :-)

Cheers,
Don

Comment 17 Alex Williamson 2011-09-26 20:10:50 UTC
(In reply to comment #16)
> 
> There isn't much we can do except ask for a firmware update on the hardware or
> just blacklist the device from being used with a guest :-)

Or create a device specific reset for it if we can figure out how to make it play nicely (though we can't really protect if userspace decides to do a D0->D3hot->D0 on it's own, which is all it takes to hit this).


Note You need to log in before you can comment on or make changes to this bug.