Bug 682110

Summary: kdump dont't work on megaraid_sas
Product: Red Hat Enterprise Linux 6 Reporter: Chao Ye <cye>
Component: kernelAssignee: Tomas Henzl <thenzl>
Status: CLOSED ERRATA QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: ajb2, bugproxy, czhang, emcnabb, haruo.tomita, jjarvis, jkachuck, phan, qcai, rlerch
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.32-130.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 692099 (view as bug list) Environment:
Last Closed: 2011-05-19 12:27:49 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 684385, 692099    
Attachments:
Description Flags
serial console log
none
sosreport
none
Patch to reset msix in megaraid_sas driver in kdump kernel
none
console log
none
serial console log none

Description Chao Ye 2011-03-04 07:20:32 UTC 
Description of problem:
[root@dell-pet310-02 ~]# rpm -q kernel kexec-tools 
kernel-2.6.32-119.el6.x86_64
kexec-tools-2.0.0-167.el6.x86_64
[root@dell-pet310-02 ~]# touch /etc/kdump.conf 
[root@dell-pet310-02 ~]# service kdump restart
Stopping kdump:                                            [  OK  ]
Detected change(s) the following file(s):
  
  /etc/kdump.conf
Rebuilding /boot/initrd-2.6.32-119.el6.x86_64kdump.img
Your running kernel is using more than 70% of the amount of space you reserved for kdump, you should consider increasing your crashkerne[WARNING]tion
Starting kdump:                                            [  OK  ]
[root@dell-pet310-02 ~]# tail /etc/kdump.conf 
#core_collector cp --sparse=always
#extra_bins /bin/cp
#link_delay 60
#kdump_post /var/crash/scripts/kdump-post.sh
#extra_bins /usr/bin/lftp
#disk_timeout 30
#extra_modules gfs2
#options modulename options
#default shell

[root@dell-pet310-02 ~]# echo c > /proc/sysrq-trigger
-----------------------------------------------------------
dell-pet310-02.lab.bos.redhat.com login: readahead-collector: starting delayed
readahead-collector: sorting
readahead-collector: finished
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney
SysRq : Trigger a crash
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0x20
PGD 23220a067 PUD 234e40067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/firmware/memmap/7/type
CPU 4 
Modules linked in: autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log power_]

Modules linked in: autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log power_]
Pid: 5485, comm: bash Not tainted 2.6.32-119.el6.x86_64 #1 PowerEdge T310
RIP: 0010:[<ffffffff81318fe6>]  [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0
RSP: 0018:ffff880232d7de18  EFLAGS: 00010096
RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000000c99
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
RBP: ffff880232d7de18 R08: ffffffff81b9de20 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000001 R12: 0000000000000000
R13: ffffffff81a99a40 R14: 0000000000000286 R15: 0000000000000007
FS:  00007f560c1c7700(0000) GS:ffff88002f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000232dca000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 5485, threadinfo ffff880232d7c000, task ffff8802318974c0)
Stack:
 ffff880232d7de68 ffffffff813192a2 ffff8802318974c0 ffff880200000000
<0> 0000000d80df4018 0000000000000002 ffff8802342f7e00 00007f560c1cf000
<0> 0000000000000002 fffffffffffffffb ffff880232d7de98 ffffffff8131935e
Call Trace:
 [<ffffffff813192a2>] __handle_sysrq+0x132/0x1a0
 [<ffffffff8131935e>] write_sysrq_trigger+0x4e/0x50
 [<ffffffff811d33fe>] proc_reg_write+0x7e/0xc0
 [<ffffffff81170c58>] vfs_write+0xb8/0x1a0
 [<ffffffff810d1652>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff81171691>] sys_write+0x51/0x90
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
Code: d0 88 81 23 81 f0 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 
RIP  [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0x20
 RSP <ffff880232d7de18>
CR2: 0000000000000000
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.32-119.el6.x86_64 (mockbuild.bos.redhat.com) 1
Command line: ro root=/dev/mapper/vg_dellpet31002-lv_root rd_LVM_LV=vg_dellpeK
KERNEL supported cpus:
  Intel GenuineIntel
  AMD AuthenticAMD
  Centaur CentaurHauls
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000100 - 0000000000099000 (usable)
 BIOS-e820: 0000000000100000 - 00000000bf699000 (usable)
 BIOS-e820: 00000000bf699000 - 00000000bf6af000 (reserved)
 BIOS-e820: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
 BIOS-e820: 00000000bf6ce000 - 00000000c0000000 (reserved)
 BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
 BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved)
 BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
last_pfn = 0x240000 max_arch_pfn = 0x400000000
user-defined physical RAM map:
 user: 0000000000000000 - 00000000000a0000 (usable)
 user: 00000000020a0000 - 0000000009ffb000 (usable)
 user: 00000000bf699000 - 00000000bf6af000 (reserved)
 user: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
 user: 00000000bf6ce000 - 00000000c0000000 (reserved)
 user: 00000000e0000000 - 00000000f0000000 (reserved)
 user: 00000000fe000000 - 0000000100000000 (reserved)
DMI 2.6 present.
last_pfn = 0x9ffb max_arch_pfn = 0x400000000
x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106
init_memory_mapping: 0000000000000000-0000000009ffb000
RAMDISK: 09a2e000 - 09fee12f
ACPI: RSDP 00000000000f0a80 00024 (v02 DELL  )
ACPI: XSDT 00000000000f0b80 00094 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: FACP 00000000bf6c3bb4 000F4 (v03 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: DSDT 00000000bf6af000 03E43 (v01 DELL   PE_SC3   00000001 INTL 20050624)
ACPI: FACS 00000000bf6c6000 00040
ACPI: APIC 00000000bf6c3478 00152 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SPCR 00000000bf6c35cc 00050 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: HPET 00000000bf6c3620 00038 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: DM__ 00000000bf6c365c 000A8 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: MCFG 00000000bf6c3850 0003C (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: WD__ 00000000bf6c3890 00134 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SLIC 00000000bf6c39c8 00024 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: ERST 00000000bf6b2fc4 00270 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: HEST 00000000bf6b3234 003A8 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: BERT 00000000bf6b2e44 00030 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: EINJ 00000000bf6b2e74 00150 (v01 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: TCPA 00000000bf6c3b4c 00064 (v02 DELL   PE_SC3   00000001 DELL 00000001)
ACPI: SSDT 00000000bf6c7000 02784 (v01  INTEL PPM RCM  80000001 INTL 20061109)
No NUMA configuration found
Faking a node at 0000000000000000-0000000009ffb000
Bootmem setup node 0 0000000000000000-0000000009ffb000
  NODE_DATA [0000000000009000 - 000000000003cfff]
  bootmap [000000000003d000 -  000000000003e3ff] pages 2
(7 early reservations) ==> bootmem [0000000000 - 0009ffb000]
  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
  #2 [0003000000 - 0003f453a4]    TEXT DATA BSS ==> [0003000000 - 0003f453a4]
  #3 [0009a2e000 - 0009fee12f]          RAMDISK ==> [0009a2e000 - 0009fee12f]
  #4 [0000099800 - 0000100000]    BIOS reserved ==> [0000099800 - 0000100000]
  #5 [0003f46000 - 0003f46459]              BRK ==> [0003f46000 - 0003f46459]
  #6 [0000008000 - 0000009000]          PGTABLE ==> [0000008000 - 0000009000]
found SMP MP-table at [ffff8800000fe710] fe710
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   0x00100000 -> 0x00100000
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x000020a0 -> 0x00009ffb
ACPI: PM-Timer IO Port: 0x808
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled)
ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled)
ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled)
ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled)
ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled)
ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled)
ACPI: LAPIC (acpi_id[0x09] lapic_id[0x28] disabled)
ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x29] disabled)
ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] disabled)
ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2b] disabled)
ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2c] disabled)
ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2d] disabled)
ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] disabled)
ACPI: LAPIC (acpi_id[0x10] lapic_id[0x2f] disabled)
ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled)
ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled)
ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled)
ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled)
ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled)
ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled)
ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled)
ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled)
ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled)
ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled)
ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled)
ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled)
ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled)
ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled)
ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled)
ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled)
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x8086a701 base: 0xfed00000
SMP: Allowing 32 CPUs, 24 hotplug CPUs
PM: Registered nosave memory: 00000000000a0000 - 00000000020a0000
Allocating PCI resources starting at 9ffb000 (gap: 9ffb000:b569e000)
Booting paravirtualized kernel on bare hardware
NR_CPUS:4096 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:1
PERCPU: Embedded 30 pages/cpu @ffff880002a00000 s92440 r8192 d22248 u131072
pcpu-alloc: s92440 r8192 d22248 u131072 alloc=1*2097152
pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 
pcpu-alloc: [0] 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 32202
Policy zone: DMA32
Kernel command line: ro root=/dev/mapper/vg_dellpet31002-lv_root rd_LVM_LV=vgK
Misrouted IRQ fixup and polling support enabled
This may significantly impact system performance
Disabling memory control group subsystem
PID hash table entries: 512 (order: 0, 4096 bytes)
Checking aperture...
No AGP bridge found
AMD-Vi disabled by default: pass amd_iommu=on to enable
Memory: 101076k/163820k available (5005k kernel code, 32772k absent, 29972k r)
Hierarchical RCU implementation.
NR_IRQS:33024 nr_irqs:664
Extended CMOS year: 2000
do_IRQ: 0.105 No irq handler for vector (irq -1)
Console: colour VGA+ 80x25
console [ttyS1] enabled
HPET: 8 timers in total, 5 timers will be used for per-cpu timer
Fast TSC calibration using PIT
Detected 2926.091 MHz processor.
Calibrating delay loop (skipped), value calculated using timer frequency.. 58)
pid_max: default: 32768 minimum: 301
Security Framework initialized
SELinux:  Initializing.
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Mount-cache hash table entries: 256
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys devices
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
Initializing cgroup subsys blkio
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
mce: CPU supports 9 MCE banks
using mwait in idle threads.
SMP alternatives: switching to UP code
ACPI: Core revision 20090903
ftrace: converting mcount calls to 0f 1f 44 00 00
ftrace: allocating 20654 entries in 81 pages
Setting APIC routing to physical flat
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
CPU0: Intel(R) Xeon(R) CPU           X3470  @ 2.93GHz stepping 05
Performance Events: PEBS fmt1+, Nehalem events, Broken BIOS detected, using s.
[Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 186 is 53003c)
NMI watchdog disabled for cpu0: unable to create perf event: -2
Brought up 1 CPUs
Total of 1 processors activated (5852.18 BogoMIPS).
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
ACPI: bus type pci registered
PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
PCI: MCFG area at e0000000 reserved in E820
PCI: Using MMCONFIG at e0000000 - efffffff
PCI: Using configuration type 1 for base access
bio: create slab <bio-0> at 0
ACPI: BIOS _OSI(Linux) query ignored
ACPI: Interpreter enabled
ACPI: (supports S0 S4 S5)
ACPI: Using IOAPIC for interrupt routing
ACPI: No dock devices found.
ACPI: PCI Root Bridge [PCI0] (0000:00)
pci 0000:00:00.0: PME# supported from D0 D3hot D3cold
pci 0000:00:00.0: PME# disabled
pci 0000:00:03.0: PME# supported from D0 D3hot D3cold
pci 0000:00:03.0: PME# disabled
pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
pci 0000:00:05.0: PME# disabled
pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1a.0: PME# disabled
pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.0: PME# disabled
pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.2: PME# disabled
pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.3: PME# disabled
pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
pci 0000:00:1c.4: PME# disabled
pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold
pci 0000:00:1d.0: PME# disabled
pci 0000:04:00.0: PME# supported from D0 D3hot
pci 0000:04:00.0: PME# disabled
pci 0000:04:00.1: PME# supported from D0 D3hot
pci 0000:04:00.1: PME# disabled
pci 0000:02:00.0: PME# supported from D0 D3hot D3cold
pci 0000:02:00.0: PME# disabled
pci 0000:02:00.1: PME# supported from D0 D3hot D3cold
pci 0000:02:00.1: PME# disabled
pci 0000:06:00.0: PME# supported from D0 D3hot D3cold
pci 0000:06:00.0: PME# disabled
pci 0000:06:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it w'
pci 0000:07:00.0: PME# supported from D0 D3hot D3cold
pci 0000:07:00.0: PME# disabled
pci 0000:07:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it w'
pci 0000:05:00.0: PME# supported from D0 D3hot
pci 0000:05:00.0: PME# disabled
pci 0000:05:00.1: PME# supported from D0 D3hot
pci 0000:05:00.1: PME# disabled
pci 0000:00:1e.0: transparent bridge
ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
HEST: HEST table parsing is initialized.
vgaarb: device added: PCI:0000:01:03.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
PCI: Using ACPI for IRQ routing
NetLabel: Initializing
NetLabel:  domain hash size = 128
NetLabel:  protocols = UNLABELED CIPSOv4
NetLabel:  unlabeled traffic allowed by default
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 24, 25, 26, 27, 28, 0
hpet0: 8 comparators, 64-bit 14.318180 MHz counter
Switching to clocksource tsc
pnp: PnP ACPI init
ACPI: bus type pnp registered
pnp: PnP ACPI: found 13 devices
ACPI: ACPI bus type pnp unregistered
system 00:07: ioport range 0x800-0x87f has been reserved
system 00:07: ioport range 0x880-0x8ff has been reserved
system 00:07: ioport range 0x900-0x91f has been reserved
system 00:07: ioport range 0x920-0x923 has been reserved
system 00:07: ioport range 0x924-0x924 has been reserved
system 00:07: ioport range 0xca0-0xca7 has been reserved
system 00:07: ioport range 0xca9-0xcab has been reserved
system 00:07: ioport range 0xcad-0xcaf has been reserved
system 00:08: ioport range 0xca8-0xca8 has been reserved
system 00:08: ioport range 0xcac-0xcac has been reserved
system 00:0a: iomem range 0xe0000000-0xefffffff has been reserved
system 00:0c: iomem range 0xfed50000-0xfed51fff has been reserved
pci 0000:04:00.1: BAR 6: address space collision on of device [0xdf100000-0xd]
pci 0000:00:03.0: PCI bridge to [bus 04-04]
pci 0000:00:03.0: PCI bridge, secondary bus 0000:04
pci 0000:00:03.0:   bridge window [io  disabled]
pci 0000:00:03.0:   bridge window [0xdf100000-0xdf2fffff]
pci 0000:00:03.0:   bridge window [0xc0000000-0xc00fffff]
pci 0000:00:05.0: PCI bridge to [bus 03-03]
pci 0000:00:05.0: PCI bridge, secondary bus 0000:03
pci 0000:00:05.0:   bridge window [0xf000-0xffff]
pci 0000:00:05.0:   bridge window [0xdf300000-0xdf3fffff]
pci 0000:00:05.0:   bridge window [mem pref disabled]
pci 0000:00:1c.0: PCI bridge to [bus 02-02]
pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02
pci 0000:00:1c.0:   bridge window [io  disabled]
pci 0000:00:1c.0:   bridge window [0xda000000-0xddffffff]
pci 0000:00:1c.0:   bridge window [mem pref disabled]
pci 0000:00:1c.2: PCI bridge to [bus 06-06]
pci 0000:00:1c.2: PCI bridge, secondary bus 0000:06
pci 0000:00:1c.2:   bridge window [0xe000-0xefff]
pci 0000:00:1c.2:   bridge window [0xdf400000-0xdf4fffff]
pci 0000:00:1c.2:   bridge window [mem pref disabled]
pci 0000:00:1c.3: PCI bridge to [bus 07-07]
pci 0000:00:1c.3: PCI bridge, secondary bus 0000:07
pci 0000:00:1c.3:   bridge window [0xd000-0xdfff]
pci 0000:00:1c.3:   bridge window [0xdf500000-0xdf5fffff]
pci 0000:00:1c.3:   bridge window [mem pref disabled]
pci 0000:00:1c.4: PCI bridge to [bus 05-05]
pci 0000:00:1c.4: PCI bridge, secondary bus 0000:05
pci 0000:00:1c.4:   bridge window [io  disabled]
pci 0000:00:1c.4:   bridge window [mem disabled]
pci 0000:00:1c.4:   bridge window [0xd7000000-0xd97fffff]
pci 0000:00:1e.0: PCI bridge to [bus 01-01]
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:01
pci 0000:00:1e.0:   bridge window [io  disabled]
pci 0000:00:1e.0:   bridge window [0xde000000-0xdeffffff]
pci 0000:00:1e.0:   bridge window [0xd9800000-0xd9ffffff]
pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
pci 0000:00:05.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 1, 8192 bytes)
TCP established hash table entries: 4096 (order: 4, 65536 bytes)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
Trying to unpack rootfs image as initramfs...
Freeing initrd memory: 5888k freed
audit: initializing netlink socket (disabled)
type=2000 audit(1299204427.598:1): initialized
HugeTLB registered 2 MB page size, pre-allocated 0 pages
VFS: Disk quotas dquot_6.5.2
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
msgmni has been set to 208
alg: No test for stdrng (krng)
ksign: Installing public key data
Loading keyring
- Added public key 8CDF1AE31685CBAC
- User ID: Red Hat, Inc. (Kernel Module GPG key)
- Added public key D4A26C9CCD09BEDA
- User ID: Red Hat Enterprise Linux Driver Update Program <secalert>
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
pciehp: PCI Express Hot Plug Controller Driver version: 0.4
acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
pci-stub: invalid id string ""
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: Power Button [PWRF]
processor LNXCPU:00: registered as cooling_device0
processor LNXCPU:01: registered as cooling_device1
processor LNXCPU:02: registered as cooling_device2
processor LNXCPU:03: registered as cooling_device3
processor LNXCPU:04: registered as cooling_device4
processor LNXCPU:05: registered as cooling_device5
processor LNXCPU:06: registered as cooling_device6
processor LNXCPU:07: registered as cooling_device7
ERST: Error Record Serialization Table (ERST) support is initialized.
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
�serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input1
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ehci_hcd 0000:00:1a.0: EHCI Host Controller
ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.0: debug port 2
ehci_hcd 0000:00:1a.0: irq 22, io mem 0xdf0fc000
ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-119.el6.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.0
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ehci_hcd 0000:00:1d.0: EHCI Host Controller
ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.0: debug port 2
ehci_hcd 0000:00:1d.0: irq 22, io mem 0xdf0fe000
ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-119.el6.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.0
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos 00:04: RTC can wake from S4
rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one day, y3k, 242 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:04: setting system clock to 2011-03-04 02:07:10 UTC (1299204430)
Initalizing network drop monitor service
Freeing unused kernel memory: 1228k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1120k freed
Freeing unused kernel memory: 1800k freed
Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Free memory/Total memory (free %): 75028 / 111116 ( 67.5222 )
Loading jbd2.ko module
Loading mbcache.ko module
Loading ext4.ko module
Loading crc-t10dif.ko module
Loading sd_mod.ko module
Loading ata_generic.ko module
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel
Loading dm-log.ko module
Loading dm-region-hash.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
Loading autofs4.ko module
Loading sunrpc.ko module
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Loading ipv6.ko module
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Loading hwmon.ko module
Loading libcrc32c.ko module
Loading mdio.ko module
Loading e1000e.ko module
e1000e: Intel(R) PRO/1000 Network Driver - 1.2.20-k2
e1000e: Copyright (c) 1999 - 2010 Intel Corporation.
e1000e 0000:06:00.0: Disabling ASPM  L1
e1000e 0000:06:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
usb 1-1: new high speed USB device using ehci_hcd and address 2
e1000e 0000:06:00.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:15:17:ef:95:29
e1000e 0000:06:00.0: eth0: Intel(R) PRO/1000 Network Connection
e1000e 0000:06:00.0: eth0: MAC: 1, PHY: 4, PBA No: D28777-005
e1000e 0000:07:00.0: Disabling ASPM  L1
e1000e 0000:07:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
usb 1-1: New USB device found, idVendor=8087, idProduct=0020
usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1: configuration #1 chosen from 1 choice
hub 1-1:1.0: USB hub found
hub 1-1:1.0: 6 ports detected
usb 2-1: new high speed USB device using ehci_hcd and address 2
e1000e 0000:07:00.0: eth1: (PCI Express:2.5GB/s:Width x1) 00:15:17:ef:10:e9
e1000e 0000:07:00.0: eth1: Intel(R) PRO/1000 Network Connection
e1000e 0000:07:00.0: eth1: MAC: 1, PHY: 4, PBA No: D28777-005
Loading bnx2.ko module
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.17 (July 18, 20)
bnx2 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
bnx2 0000:02:00.0: pci_enable_pcie_error_reporting failed 0xfffffffb
bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-mips-09-6.0.17.fw
bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Ex2
bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
bnx2 0000:02:00.1: pci_enable_pcie_error_reporting failed 0xfffffffb
bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-mips-09-6.0.17.fw
bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw
bnx2 0000:02:00.1: eth3: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Ex3
Loading microcodusb 2-1: New USB device found, idVendor=8087, idProduct=0020
e.ko module
usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
microcode: CPU0 sig=0x106e5, pf=0x2, revision=0x4
platform microcode: firmware: requesting intel-ucode/06-1e-05
usb 2-1: configuration #1 chosen from 1 choice
hub 2-1:1.0: USB hub found
hub 2-1:1.0: 8 ports detected
usb 1-1.1: new high speed USB device using ehci_hcd and address 3
usb 1-1.1: New USB device found, idVendor=0424, idProduct=2514
usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
usb 1-1.1: configuration #1 chosen from 1 choice
hub 1-1.1:1.0: USB hub found
hub 1-1.1:1.0: 2 ports detected
Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
Loading dcdbas.ko module
dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
Loading hed.ko module
Loading iTCO_vendor_support.ko miTCO_vendor_support: vendor-support=0
odule
Loading shpchp.ko module
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Loading sg.ko module
Loading bna.ko module
Brocade 10G Ethernet driver
bnad_pci_probe : (0xffff8800088f2000, 0xffffffffa02a7720) PCI Func : (1)
bna 0000:04:00.1: firmware: requesting ctfw_cna.bin
bna 0000:04:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16
bar0 mapped to ffffc90010580000, len 262144
bna: GRO enabled, using kernel stack GRO
Loading cdrom.ko module
Loading pata_acpi.ko module
pata_acpi 0000:00:1f.2: PCI INT A -> GSI 20 (level, low) -> IRQ 20
pata_acpi 0000:00:1f.2: PCI INT A disabled
pata_acpi 0000:00:1f.5: PCI INT B -> GSI 21 (level, low) -> IRQ 21
pata_acpi 0000:00:1f.5: PCI INT B disabled
Loading ata_piix.ko module
ata_piix 0000:00:1f.2: PCI INT A -> GSI 20 (level, low) -> IRQ 20
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0xcca0 ctl 0xcc90 bmdma 0xccc0 irq 20
ata2: SATA max UDMA/133 cmd 0xcca8 ctl 0xcc94 bmdma 0xccc8 irq 20
ata_piix 0000:00:1f.5: PCI INT B -> GSI 21 (level, low) -> IRQ 21
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
scsi2 : ata_piix
scsi3 : ata_piix
ata3: SATA max UDMA/133 cmd 0xccb0 ctl 0xcc98 bmdma 0xcce0 irq 21
ata4: SATA max UDMA/133 cmd 0xccb8 ctl 0xcc9c bmdma 0xcce8 irq 21
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata3.00: ATAPI: PLDS DVD-ROM DH-16D5S, VD15, max UDMA/100
ata3.00: configured for UDMA/100
ata2.00: failed to resume link (SControl 0)
ata4: failed to resume link (SControl 0)
ata4: SATA link down (SStatus 4 SControl 0)
ata1.01: failed to resume link (SControl 0)
ata1.00: SATA link down (SStatus 4 SControl 300)
ata1.01: SATA link down (SStatus 4 SControl 0)
ata2.01: failed to resume link (SControl 0)
ata2.00: SATA link down (SStatus 4 SControl 0)
ata2.01: SATA link down (SStatus 4 SControl 0)
scsi 2:0:0:0: CD-ROM            PLDS     DVD-ROM DH-16D5S VD15 PQ: 0 ANSI: 5
scsi 2:0:0:0: Attached scsi generic sg0 type 5
Loading megaraid_sas.ko module
megasas: 00.00.05.29-RH1 Tue. Dec. 7 17:00:00 PDT 2010
megasas: 0x1000:0x0079:0x1028:0x1f16: bus 3:slot 0:func 0
megaraid_sas 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
megasas: Waiting for FW to come to ready state
megasas: FW now in Ready state
megasas_init_mfi: fw_support_ieee=67108864
megasas: INIT adapter done
megaraid_sas: fw state:c0000000
megasas: fwstate:c0000000, dis_OCR=0
INFO: task insmod:230 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   230      1 0x00000000
 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac
 ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8
Call Trace:
 [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f257>] local_pci_probe+0x17/0x20
 [<ffffffff81280441>] pci_device_probe+0x101/0x120
 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0
 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0
 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339afe>] driver_attach+0x1e/0x20
 [<ffffffff81339400>] bus_add_driver+0x200/0x300
 [<ffffffff8133a336>] driver_register+0x76/0x140
 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b



INFO: task insmod:230 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   230      1 0x00000000
 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac
 ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8
Call Trace:
 [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f257>] local_pci_probe+0x17/0x20
 [<ffffffff81280441>] pci_device_probe+0x101/0x120
 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0
 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0
 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339afe>] driver_attach+0x1e/0x20
 [<ffffffff81339400>] bus_add_driver+0x200/0x300
 [<ffffffff8133a336>] driver_register+0x76/0x140
 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
INFO: task insmod:230 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   230      1 0x00000000
 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac
 ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8
Call Trace:
 [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f257>] local_pci_probe+0x17/0x20
 [<ffffffff81280441>] pci_device_probe+0x101/0x120
 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0
 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0
 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339afe>] driver_attach+0x1e/0x20
 [<ffffffff81339400>] bus_add_driver+0x200/0x300
 [<ffffffff8133a336>] driver_register+0x76/0x140
 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b

Version-Release number of selected component (if applicable):
kernel-2.6.32-119.el6.x86_64
kexec-tools-2.0.0-167.el6.x86_64

How reproducible:
issue found on dell-pet310-02.lab.bos.redhat.com

Steps to Reproduce:
1.Install RHEL6.1-20110224.2
2.Upgrade kernel/kexec-tools
3.Trigger crash
  
Actual results:
System got hang

Expected results:
vmcore saved

Additional info:
Another beaker test:
https://beaker.engineering.redhat.com/recipes/119932
Comment 2 Chao Ye 2011-03-15 06:42:50 UTC 
Same issue found on intel-s3e36-01.rhts.eng.nay.redhat.com when trigger crash via NMI button:
=========================================================
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:09: registered as cooling_device9
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0a: registered as cooling_device10
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0b: registered as cooling_device11
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0c: registered as cooling_device12
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0d: registered as cooling_device13
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0e: registered as cooling_device14
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:0f: registered as cooling_device15
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:10: registered as cooling_device16
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:11: registered as cooling_device17
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:12: registered as cooling_device18
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:13: registered as cooling_device19
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:14: registered as cooling_device20
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:15: registered as cooling_device21
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:16: registered as cooling_device22
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:17: registered as cooling_device23
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:18: registered as cooling_device24
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:19: registered as cooling_device25
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1a: registered as cooling_device26
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1b: registered as cooling_device27
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1c: registered as cooling_device28
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1d: registered as cooling_device29
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1e: registered as cooling_device30
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:1f: registered as cooling_device31
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:20: registered as cooling_device32
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:21: registered as cooling_device33
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:22: registered as cooling_device34
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:23: registered as cooling_device35
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:24: registered as cooling_device36
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:25: registered as cooling_device37
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:26: registered as cooling_device38
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:27: registered as cooling_device39
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:28: registered as cooling_device40
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:29: registered as cooling_device41
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2a: registered as cooling_device42
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2b: registered as cooling_device43
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2c: registered as cooling_device44
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2d: registered as cooling_device45
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2e: registered as cooling_device46
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:2f: registered as cooling_device47
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:30: registered as cooling_device48
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:31: registered as cooling_device49
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:32: registered as cooling_device50
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:33: registered as cooling_device51
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:34: registered as cooling_device52
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:35: registered as cooling_device53
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:36: registered as cooling_device54
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:37: registered as cooling_device55
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:38: registered as cooling_device56
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:39: registered as cooling_device57
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3a: registered as cooling_device58
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3b: registered as cooling_device59
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3c: registered as cooling_device60
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3d: registered as cooling_device61
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3e: registered as cooling_device62
ACPI: HARDWARE addr space,NOT supported yet
processor LNXCPU:3f: registered as cooling_device63
ERST: Error Record Serialization Table (ERST) support is initialized.
Non-volatile memory driver v1.3
Linux agpgart interface v0.103
tpm_tis 00:0a: 1.2 TPM (device-id 0x4A10, rev-id 78)
crash memory driver: version 1.1
Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
�serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
brd: module loaded
loop: module loaded
input: Macintosh mouse button emulation as /devices/virtual/input/input2
Fixed MDIO Bus: probed
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 18 (level, low) -> IRQ 18
ehci_hcd 0000:00:1a.7: EHCI Host Controller
ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1a.7: debug port 1
ehci_hcd 0000:00:1a.7: irq 18, io mem 0x9bc01000
ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: EHCI Host Controller
usb usb1: Manufacturer: Linux 2.6.32-122.el6.x86_64 ehci_hcd
usb usb1: SerialNumber: 0000:00:1a.7
usb usb1: configuration #1 chosen from 1 choice
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 6 ports detected
ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 23 (level, low) -> IRQ 23
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: irq 23, io mem 0x9bc00000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00
usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: EHCI Host Controller
usb usb2: Manufacturer: Linux 2.6.32-122.el6.x86_64 ehci_hcd
usb usb2: SerialNumber: 0000:00:1d.7
usb usb2: configuration #1 chosen from 1 choice
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 6 ports detected
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
uhci_hcd: USB Universal Host Controller Interface driver
uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
uhci_hcd 0000:00:1a.0: UHCI Host Controller
uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1a.0: irq 16, io base 0x000060c0
usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: UHCI Host Controller
usb usb3: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb3: SerialNumber: 0000:00:1a.0
usb usb3: configuration #1 chosen from 1 choice
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21
uhci_hcd 0000:00:1a.1: UHCI Host Controller
uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1a.1: irq 21, io base 0x000060a0
usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb4: Product: UHCI Host Controller
usb usb4: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb4: SerialNumber: 0000:00:1a.1
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
uhci_hcd 0000:00:1a.2: PCI INT D -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1a.2: UHCI Host Controller
uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1a.2: irq 19, io base 0x00006080
usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb5: Product: UHCI Host Controller
usb usb5: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb5: SerialNumber: 0000:00:1a.2
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
uhci_hcd 0000:00:1d.0: UHCI Host Controller
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6
uhci_hcd 0000:00:1d.0: irq 23, io base 0x00006060
usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb6: Product: UHCI Host Controller
usb usb6: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb6: SerialNumber: 0000:00:1d.0
usb usb6: configuration #1 chosen from 1 choice
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19
uhci_hcd 0000:00:1d.1: UHCI Host Controller
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7
uhci_hcd 0000:00:1d.1: irq 19, io base 0x00006040
usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb7: Product: UHCI Host Controller
usb usb7: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb7: SerialNumber: 0000:00:1d.1
usb usb7: configuration #1 chosen from 1 choice
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 2 ports detected
uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8
uhci_hcd 0000:00:1d.2: irq 18, io base 0x00006020
usb usb8: New USB device found, idVendor=1d6b, idProduct=0001
usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb8: Product: UHCI Host Controller
usb usb8: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd
usb usb8: SerialNumber: 0000:00:1d.2
usb usb8: configuration #1 chosen from 1 choice
hub 8-0:1.0: USB hub found
hub 8-0:1.0: 2 ports detected
PNP: No PS/2 controller found. Probing ports directly.
mice: PS/2 mouse device common for all mice
rtc_cmos 00:03: RTC can wake from S4
rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
cpuidle: using governor ladder
cpuidle: using governor menu
usbcore: registered new interface driver hiddev
usbcore: registered new interface driver usbhid
usbhid: v2.6:USB HID core driver
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 17
registered taskstats version 1
rtc_cmos 00:03: setting system clock to 2011-03-15 01:31:00 UTC (1300152660)
Initalizing network drop monitor service
Freeing unused kernel memory: 1228k freed
Write protecting the kernel read-only data: 10240k
Freeing unused kernel memory: 1120k freed
Freeing unused kernel memory: 1800k freed
Mounting proc fiusb 4-1: new full speed USB device using uhci_hcd and address 2
lesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Free memory/Total memory (free %): 65560 / 108284 ( 60.5445 )
Loading jbd2.ko module
Loading mbcache.ko module
Loading ext4.ko module
Loading crc-t10dif.ko module
Loading sd_mod.ko module
Loading ata_generic.ko module
Loading dm-mod.ko module
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel
Loading dm-log.ko module
Loading dm-region-hash.ko module
Loading dm-mirror.ko module
Loading dm-zero.ko module
Loading dm-snapshot.ko module
usb 4-1: New USB device found, idVendor=046b, idProduct=ff10
usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 4-1: Product: Virtual Keyboard and Mouse
usb 4-1: Manufacturer: American Megatrends Inc.
usb 4-1: SerialNumber: serial
Loading autofs4.ko module
Loading sunrpc.ko module
usb 4-1: configuration #1 chosen from 1 choice
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
Loading freq_table.ko module
input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.0/input/input3
Loading ipv6.ko module
generic-usb 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1/input0
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
Loading microcode.ko module
microcode: CPU0 sig=0x206e6, pf=0x4, revision=0x6
input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.1/input/input4
platform microcode: firmware: requesting intel-ucode/06-2e-06
generic-usb 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1/input1
usb 7-2: new low speed USB device using uhci_hcd and address 2
usb 7-2: New USB device found, idVendor=0624, idProduct=0307
usb 7-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 7-2: Product: Avocent DSRIQ-USB
usb 7-2: Manufacturer: Avocent
usb 7-2: configuration #1 chosen from 1 choice
input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input5
generic-usb 0003:0624:0307.0003: input,hidraw2: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2/input0
input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input6
generic-usb 0003:0624:0307.0004: input,hidraw3: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2/input1
Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba
Loading hed.ko module
Loading i2c-core.ko module
Loading iTCO_vendor_support.ko module
iTCO_vendor_support: vendor-support=0
Loading edac_core.ko module
EDAC MC: Ver: 2.1.0 Mar  9 2011
Loading sg.ko module
Loading dca.ko module
dca service started, version 1.12.1
Loading cdrom.ko module
Loading pata_acpi.ko module
pata_acpi 0000:00:1f.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19
pata_acpi 0000:00:1f.2: PCI INT A disabled
pata_acpi 0000:00:1f.5: PCI INT D -> GSI 21 (level, low) -> IRQ 21
pata_acpi 0000:00:1f.5: PCI INT D disabled
Loading ata_piix.ko module
ata_piix 0000:00:1f.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19
ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
scsi0 : ata_piix
scsi1 : ata_piix
ata1: SATA max UDMA/133 cmd 0x6138 ctl 0x614c bmdma 0x6110 irq 19
ata2: SATA max UDMA/133 cmd 0x6130 ctl 0x6148 bmdma 0x6118 irq 19
ata_piix 0000:00:1f.5: PCI INT D -> GSI 21 (level, low) -> IRQ 21
ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
scsi2 : ata_piix
scsi3 : ata_piix
ata3: SATA max UDMA/133 cmd 0x6128 ctl 0x6144 bmdma 0x60f0 irq 21
ata4: SATA max UDMA/133 cmd 0x6120 ctl 0x6140 bmdma 0x60f8 irq 21
ata4: SATA link down (SStatus 0 SControl 300)
ata3: SATA link down (SStatus 4 SControl 300)
ata2.00: SATA link down (SStatus 0 SControl 300)
ata2.01: SATA link down (SStatus 0 SControl 300)
ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.01: SATA link down (SStatus 0 SControl 300)
ata1.00: ATAPI: Optiarc DVD RW AD-7580S, FX04, max UDMA/100
ata1.00: configured for UDMA/100
scsi 0:0:0:0: CD-ROM            Optiarc  DVD RW AD-7580S  FX04 PQ: 0 ANSI: 5
scsi 0:0:0:0: Attached scsi generic sg0 type 5
Loading megaraid_sas.ko module
megasas: 00.00.05.29-RH1 Tue. Dec. 7 17:00:00 PDT 2010
megasas: 0x1000:0x0079:0x8086:0x9261: bus 7:slot 0:func 0
megaraid_sas 0000:07:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
megasas: Waiting for FW to come to ready state
megasas: FW now in Ready state
megasas_init_mfi: fw_support_ieee=0
megasas: INIT adapter done
megaraid_sas: fw state:c0000000
megasas: fwstate:c0000000, dis_OCR=0
DRHD: handling fault status reg 102
INTR-REMAP: Request device [[01:00.0] fault index 32
INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
INFO: task insmod:201 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   201      1 0x00000000
 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc
 ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038
Call Trace:
 [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20
 [<ffffffff812806d1>] pci_device_probe+0x101/0x120
 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0
 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0
 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339d8e>] driver_attach+0x1e/0x20
 [<ffffffff81339690>] bus_add_driver+0x200/0x300
 [<ffffffff8133a5c6>] driver_register+0x76/0x140
 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
INFO: task insmod:201 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   201      1 0x00000000
 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc
 ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038
Call Trace:
 [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20
 [<ffffffff812806d1>] pci_device_probe+0x101/0x120
 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0
 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0
 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339d8e>] driver_attach+0x1e/0x20
 [<ffffffff81339690>] bus_add_driver+0x200/0x300
 [<ffffffff8133a5c6>] driver_register+0x76/0x140
 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
INFO: task insmod:201 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   201      1 0x00000000
 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc
 ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038
Call Trace:
 [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20
 [<ffffffff812806d1>] pci_device_probe+0x101/0x120
 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0
 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0
 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339d8e>] driver_attach+0x1e/0x20
 [<ffffffff81339690>] bus_add_driver+0x200/0x300
 [<ffffffff8133a5c6>] driver_register+0x76/0x140
 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
DRHD: handling fault status reg 202
INTR-REMAP: Request device [[01:00.0] fault index 33
INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
DRHD: handling fault status reg 302
INTR-REMAP: Request device [[01:00.0] fault index 35
INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
DRHD: handling fault status reg 402
INTR-REMAP: Request device [[01:00.0] fault index 31
INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
DRHD: handling fault status reg 502
INTR-REMAP: Request device [[01:00.0] fault index 37
INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear
INFO: task insmod:201 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
insmod        D 0000000000000000     0   201      1 0x00000000
 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000
 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc
 ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038
Call Trace:
 [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400
 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80
 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas]
 [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas]
 [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas]
 [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas]
 [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20
 [<ffffffff812806d1>] pci_device_probe+0x101/0x120
 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90
 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0
 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0
 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0
 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90
 [<ffffffff81339d8e>] driver_attach+0x1e/0x20
 [<ffffffff81339690>] bus_add_driver+0x200/0x300
 [<ffffffff8133a5c6>] driver_register+0x76/0x140
 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0
 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas]
 [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas]
 [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0
 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b


RHEL6.1-20110311.3 installed, system reserved for debug.
Comment 4 Qian Cai 2011-03-17 00:37:57 UTC 
*** Bug 688334 has been marked as a duplicate of this bug. ***
Comment 5 IBM Bug Proxy 2011-03-17 00:45:51 UTC 
Created attachment 485874 [details]
serial console log
Comment 6 IBM Bug Proxy 2011-03-17 00:45:58 UTC 
Created attachment 485875 [details]
sosreport
Comment 8 IBM Bug Proxy 2011-03-17 18:02:20 UTC 
------- Comment From masbock.com 2011-03-17 13:50 EDT-------
From the call traces it looks like the driver is waiting for an interrupt in response to megasas_issue_blocked_cmd. Perhaps this is an issue with MSI-x initialization in the kdump kernel.
Therefore it might be worthwhile to boot both kernels with megaraid_sas.disable_msix=1 to see if the problem goes away. (This is not meant as a workaround, only a debugging step.)
Comment 9 IBM Bug Proxy 2011-03-18 00:31:18 UTC 
------- Comment From masbock.com 2011-03-17 20:24 EDT-------
This is a snippet from the kernel messages:



It looks like megaraid_sas allocated a legacy (non-msix) interrupt vector. I'd guess that the device still thinks it's using MSI-x


The following is a vt-d error. It is the result of devices still using MSI vectors activate by the main kernel for which there are no interrupt remapping table entries. This one could have been caused by  the MSI in response to the megaraid device to the megasas_issue_blocked_cmd. (But it could also belong to the network device)
Comment 10 Qian Cai 2011-03-18 02:27:14 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Cause
    Loading megaraid_sas driver and hardware in kdump kernel.
Consequence
    Kdump kernel is unable to process any further, and insmod command is blocked with those messages.

INFO: task insmod:201 blocked for more than 120 seconds.

Workaround
    None found yet.
Comment 11 IBM Bug Proxy 2011-03-18 12:31:14 UTC 
------- Comment From shubgoya.com 2011-03-18 08:25 EDT-------
There is one more addition to the platform list affected by this issue. Below are the details:-

x3650M3 - 01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240 (rev 03)

Thx, Shubham
Comment 12 IBM Bug Proxy 2011-03-18 18:31:55 UTC 
------- Comment From masbock.com 2011-03-18 14:27 EDT-------
(In reply to comment #16)
> From the call traces it looks like the driver is waiting for an interrupt in
> response to megasas_issue_blocked_cmd. Perhaps this is an issue with MSI-x
> initialization in the kdump kernel.
> Therefore it might be worthwhile to boot both kernels with
> megaraid_sas.disable_msix=1 to see if the problem goes away. (This is not meant
> as a workaround, only a debugging step.)

The boot parameter really is megaraid_sas.msix_disable=1 (I had it backwards).

Shubham just verified that this works. This means that if the megaraid_sas driver doesn't use MSI-x the kdump kernel boots.

As a next step we should back port the following patch:

[PATCH 5/15] megaraid_sas: Fix probe_one to clear MSI-X flags in kdump
http://marc.info/?l=linux-scsi&m=129816856305895&w=2

to see if that fixes the issues.
Comment 13 IBM Bug Proxy 2011-03-21 17:31:11 UTC 
------- Comment From masbock.com 2011-03-21 13:21 EDT-------
(In reply to comment #21)
> (In reply to comment #20)
> > (In reply to comment #16)

> > As a next step we should back port the following patch:
> >
> > [PATCH 5/15] megaraid_sas: Fix probe_one to clear MSI-X flags in kdump
> > http://marc.info/?l=linux-scsi&m=129816856305895&w=2
> >
> > to see if that fixes the issues.
>
>
> Is this patch accepted in mainline? if we want this bug fixed in 6.1, we need
> to pack port it and test it asap. please give an estimate as when the pack port
> patch will be ready.
>

I thought I had seen the patch last Friday on kernel.org in 2.6.39-git8. But I don't see it anymore in git10.
Would it be possible to have somebody from LSI to comment on the status of this patch in main-line?

We can provide a back port and test results for the patch quickly.
Comment 14 IBM Bug Proxy 2011-03-21 18:01:43 UTC 
------- Comment From masbock.com 2011-03-21 13:54 EDT-------
I just verified that the patch is indeed in the latest main-line kernel git tree.
I can be found here:
http://www.kernel.org/diff/diffview.cgi?file=/pub/linux/kernel/v2.6/snapshots/patch-2.6.38-git10.bz2
Comment 15 Ryan Lerch 2011-03-21 22:56:21 UTC 
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,9 +1,3 @@
-Cause
-    Loading megaraid_sas driver and hardware in kdump kernel.
-Consequence
-    Kdump kernel is unable to process any further, and insmod command is blocked with those messages.
+Loading the megaraid_sas driver on the kdump kernel will result in the insmod command being blocked, returning messages similar to:
 
-INFO: task insmod:201 blocked for more than 120 seconds.
+INFO: task insmod:201 blocked for more than 120 seconds.-
-Workaround
-    None found yet.
Comment 16 IBM Bug Proxy 2011-03-21 23:31:05 UTC 
Created attachment 486704 [details]
Patch to reset msix in megaraid_sas driver in kdump kernel


------- Comment on attachment From masbock.com 2011-03-21 19:24 EDT-------


This is a back port of a patch that resets msix in the megaraid_sas driver in the kdump kernel.
The patch was originally posted by Adam Radford from LSI here:
http://marc.info/?l=linux-scsi&m=129816856305895&w=2
"Back port" is really an exaggeration, it's only shifted by 4 lines.

This patch depends on the "reset_devices" boot parameter being passed. This parameter is by default passed in the kdump kernel. Therefore no special documentation is needed.

It tested this patch with the linux-2.6.32-122.el6.x86_64 kernel and obtained a vmcore. The megaraid_sas driver worked fine in the kdump kernel with the patch.

This patches fixes the issue.
Comment 17 IBM Bug Proxy 2011-03-22 20:01:35 UTC 
------- Comment From tpnoonan.com 2011-03-22 15:58 EDT-------
red hat please consider as exception for rhel6.1, thnx
Comment 18 Joseph Kachuck 2011-03-22 20:11:28 UTC 
Hello,
This is requested as a blocker for 6.1.

Thank You
Joe Kachuck
Comment 21 Tomas Henzl 2011-03-28 12:15:25 UTC 
I posted the patch for review.
Comment 22 RHEL Program Management 2011-03-28 12:19:41 UTC 
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.
Comment 24 Tomas Henzl 2011-03-28 12:21:50 UTC 
Deleted Technical Notes Contents.

Old Contents:
Loading the megaraid_sas driver on the kdump kernel will result in the insmod command being blocked, returning messages similar to:

INFO: task insmod:201 blocked for more than 120 seconds.
Comment 25 IBM Bug Proxy 2011-03-29 04:11:40 UTC 
------- Comment From ramesyr1.com 2011-03-29 00:07 EDT-------
I tested the patch on Linux 2.6.32-122.el6.x86_64 kernel ,after triggering the kdump megaraid_sas driver is loading without any issues but vmcore is not generated.I could only see an empty directory under /var/crash with no vmcore in it.

FYI,
[root@mx3850x5 crash]# cd 127.0.0.1-2011-03-23-20:17:28 ;ll
total 0

I also noticed from console log that while loading SELinux policy ,the makedumpfile process is getting killed by a oom-killer while it is trying to save the vmcore.

from the above observation ,I disabled the SELinux policy and triggered the kdump but the system got hanged(console log for hang point attached), after a longtime i rebooted the machine manually and the interesting thing was it has generated the vmcore of zero size.

FYI,
----SELinux disabled----
[root@mx3850x5 ~]# getenforce
Permissive

-----Zero size vmcore-------
[root@mx3850x5 ~]# cd /var/crash/2011-03-23-21:44/ ; ll
total 0
-rw-------. 1 root root 0 Mar 23 21:44 vmcore.
Machine Type = X3850-X5
system memory = 256GB

* complete serial console log is attached.
Comment 26 IBM Bug Proxy 2011-03-29 04:22:10 UTC 
Created attachment 488322 [details]
serial console log


------- Comment (attachment only) From ramesyr1.com 2011-03-29 00:11 EDT-------
Comment 27 Tomas Henzl 2011-03-29 11:24:15 UTC 
(In reply to comment #25)
> ------- Comment From ramesyr1.com 2011-03-29 00:07 EDT-------
> I tested the patch on Linux 2.6.32-122.el6.x86_64 kernel ,after triggering the
> kdump megaraid_sas driver is loading without any issues but vmcore is not
> generated.I could only see an empty directory under /var/crash with no vmcore
> in it.
> system memory = 256GB

This look like another issue, please try, before we open another bug report, to reserve a larger memory size for the kdump kernel. It can happen is some corner cases that default value isn't enough. Try some larger values, the kernel option is crashkernel, so for example crashkernel=256M@256M or higher.
Comment 28 IBM Bug Proxy 2011-03-29 12:41:25 UTC 
------- Comment From ramesyr1.com 2011-03-29 08:35 EDT-------
(In reply to comment #32)
>
> This look like another issue, please try, before we open another bug report, to
> reserve a larger memory size for the kdump kernel. It can happen is some corner
> cases that default value isn't enough. Try some larger values, the kernel
> option is crashkernel, so for example crashkernel=256M@256M or higher.

I tried with crashkernel=512M ,the issue goes away however the vmcore
genearation fails with crashkernel=auto .
But according to the kdump documentation at Documentation/kdump/kdump.txt from
RHEL6, the crashkernel=auto reserves memory according to following table:

Memory size           Reserved memory
===========     ===============
[4G, 12G)                  256M
[12G, 128G)               512M
[128G, 256G)             768M
[256G, 378G)             1024M
[378G, 512G)             1536M
[512G, 768G)             2048M
[768G, )                    3072M

According to the above table 1024M should have been reserved for crashkernel
when specified crashkernel=auto but it is reserving only 129M.
Since the auto feature does not work as expected in kdump documentation, this
looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a
separate BUG for this issue?
Comment 29 Tomas Henzl 2011-03-29 12:58:28 UTC 
(In reply to comment #28)
> Since the auto feature does not work as expected in kdump documentation, this
> looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a
> separate BUG for this issue?

Yes please, open a new Bug for this issue.
Comment 30 Alan Brown 2011-03-29 18:53:12 UTC 
I'm seeing the same problem on RHEL 5.6
Comment 31 Tomas Henzl 2011-04-05 15:23:21 UTC 
(In reply to comment #30)
> I'm seeing the same problem on RHEL 5.6

Do you mean the msi-x issue or the problem with the size of the crashkernel?
Comment 32 Alan Brown 2011-04-06 10:04:06 UTC 
The msi-x issue.
Comment 33 Tomas Henzl 2011-04-06 11:52:26 UTC 
(In reply to comment #32)
> The msi-x issue.

Let us continue this RHEL5 issue here -> bz#692099#c5
Comment 34 IBM Bug Proxy 2011-04-06 13:42:03 UTC 
------- Comment From iranna.ankad.com 2011-04-06 09:39 EDT-------
(In reply to comment #35)
> (In reply to comment #28)
> > Since the auto feature does not work as expected in kdump documentation, this
> > looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a
> > separate BUG for this issue?
>
> Yes please, open a new Bug for this issue.

Hello Red Hat FYI,
We have opened below bugzilla to track this new issue.

Bug 71062   -  RH692764 :crashkernel=auto feature does not work as per the kdump documentation

Now, coming back to the original issue for which we are using this bugzilla (i.e kdump hang on megaraid),
the patch which we have submitted/tested is still under your review. Please let us know further status on that.

Thanks!
Comment 35 Tomas Henzl 2011-04-06 14:19:51 UTC 
(In reply to comment #34)
> Hello Red Hat FYI,
> We have opened below bugzilla to track this new issue.
> 
> Bug 71062   -  RH692764 :crashkernel=auto feature does not work as per the
> kdump documentation
I hope I added the right person to this bug.

> Now, coming back to the original issue for which we are using this bugzilla
> (i.e kdump hang on megaraid),
> the patch which we have submitted/tested is still under your review. Please let
> us know further status on that.

The patch is in POSTED state, it will updated accordingly when the patch gets included.
Comment 36 John Jarvis 2011-04-07 14:13:21 UTC 
This fix is approved and planned for inclusion in snapshot 3.
Comment 37 Aristeu Rozanski 2011-04-07 14:14:57 UTC 
Patch(es) available on kernel-2.6.32-130.el6
Comment 39 IBM Bug Proxy 2011-04-08 10:53:12 UTC 
------- Comment From iranna.ankad.com 2011-04-08 06:43 EDT-------
(In reply to comment #42)
> This fix is approved and planned for inclusion in snapshot 3.
> Patch(es) available on kernel-2.6.32-130.el6

Red Hat,
Thanks for the updates. We will retest once snap3 is available.
Comment 41 IBM Bug Proxy 2011-04-14 11:52:26 UTC 
------- Comment From ramesyr1.com 2011-04-14 07:46 EDT-------
I verified the kdump on snap3 , kdump worked fine and generated the vmcore successfully.

Thanks !
Comment 42 IBM Bug Proxy 2011-04-14 14:21:41 UTC 
------- Comment From iranna.ankad.com 2011-04-14 10:16 EDT-------
(In reply to comment #46)
> I verified the kdump on snap3 , kdump worked fine and generated the vmcore
> successfully.
>
> Thanks !

Ramesh,
Thanks for the snap3 results.

Red Hat,
We can close this bug now.  Thanks!

------- Comment From iranna.ankad.com 2011-04-14 10:16 EDT-------
Closing...
Comment 43 errata-xmlrpc 2011-05-19 12:27:49 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2011-0542.html