Bug 682110
| Summary: | kdump dont't work on megaraid_sas | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Chao Ye <cye> | ||||||||||||
| Component: | kernel | Assignee: | Tomas Henzl <thenzl> | ||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Red Hat Kernel QE team <kernel-qe> | ||||||||||||
| Severity: | high | Docs Contact: | |||||||||||||
| Priority: | high | ||||||||||||||
| Version: | 6.1 | CC: | ajb2, bugproxy, czhang, emcnabb, haruo.tomita, jjarvis, jkachuck, phan, qcai, rlerch | ||||||||||||
| Target Milestone: | rc | Keywords: | Regression | ||||||||||||
| Target Release: | --- | ||||||||||||||
| Hardware: | All | ||||||||||||||
| OS: | Linux | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | kernel-2.6.32-130.el6 | Doc Type: | Bug Fix | ||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | |||||||||||||||
| : | 692099 (view as bug list) | Environment: | |||||||||||||
| Last Closed: | 2011-05-19 12:27:49 UTC | Type: | --- | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Bug Depends On: | |||||||||||||||
| Bug Blocks: | 684385, 692099 | ||||||||||||||
| Attachments: |
|
||||||||||||||
Same issue found on intel-s3e36-01.rhts.eng.nay.redhat.com when trigger crash via NMI button: ========================================================= ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:09: registered as cooling_device9 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0a: registered as cooling_device10 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0b: registered as cooling_device11 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0c: registered as cooling_device12 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0d: registered as cooling_device13 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0e: registered as cooling_device14 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:0f: registered as cooling_device15 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:10: registered as cooling_device16 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:11: registered as cooling_device17 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:12: registered as cooling_device18 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:13: registered as cooling_device19 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:14: registered as cooling_device20 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:15: registered as cooling_device21 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:16: registered as cooling_device22 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:17: registered as cooling_device23 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:18: registered as cooling_device24 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:19: registered as cooling_device25 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1a: registered as cooling_device26 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1b: registered as cooling_device27 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1c: registered as cooling_device28 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1d: registered as cooling_device29 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1e: registered as cooling_device30 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:1f: registered as cooling_device31 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:20: registered as cooling_device32 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:21: registered as cooling_device33 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:22: registered as cooling_device34 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:23: registered as cooling_device35 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:24: registered as cooling_device36 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:25: registered as cooling_device37 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:26: registered as cooling_device38 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:27: registered as cooling_device39 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:28: registered as cooling_device40 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:29: registered as cooling_device41 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2a: registered as cooling_device42 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2b: registered as cooling_device43 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2c: registered as cooling_device44 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2d: registered as cooling_device45 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2e: registered as cooling_device46 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:2f: registered as cooling_device47 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:30: registered as cooling_device48 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:31: registered as cooling_device49 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:32: registered as cooling_device50 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:33: registered as cooling_device51 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:34: registered as cooling_device52 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:35: registered as cooling_device53 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:36: registered as cooling_device54 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:37: registered as cooling_device55 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:38: registered as cooling_device56 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:39: registered as cooling_device57 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3a: registered as cooling_device58 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3b: registered as cooling_device59 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3c: registered as cooling_device60 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3d: registered as cooling_device61 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3e: registered as cooling_device62 ACPI: HARDWARE addr space,NOT supported yet processor LNXCPU:3f: registered as cooling_device63 ERST: Error Record Serialization Table (ERST) support is initialized. Non-volatile memory driver v1.3 Linux agpgart interface v0.103 tpm_tis 00:0a: 1.2 TPM (device-id 0x4A10, rev-id 78) crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled �serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 00:08: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:09: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input2 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci_hcd 0000:00:1a.7: PCI INT C -> GSI 18 (level, low) -> IRQ 18 ehci_hcd 0000:00:1a.7: EHCI Host Controller ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:1a.7: debug port 1 ehci_hcd 0000:00:1a.7: irq 18, io mem 0x9bc01000 ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00 usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: EHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32-122.el6.x86_64 ehci_hcd usb usb1: SerialNumber: 0000:00:1a.7 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 6 ports detected ehci_hcd 0000:00:1d.7: PCI INT A -> GSI 23 (level, low) -> IRQ 23 ehci_hcd 0000:00:1d.7: EHCI Host Controller ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 2 ehci_hcd 0000:00:1d.7: debug port 1 ehci_hcd 0000:00:1d.7: irq 23, io mem 0x9bc00000 ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00 usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb2: Product: EHCI Host Controller usb usb2: Manufacturer: Linux 2.6.32-122.el6.x86_64 ehci_hcd usb usb2: SerialNumber: 0000:00:1d.7 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 6 ports detected ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver uhci_hcd 0000:00:1a.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 uhci_hcd 0000:00:1a.0: UHCI Host Controller uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1a.0: irq 16, io base 0x000060c0 usb usb3: New USB device found, idVendor=1d6b, idProduct=0001 usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb3: Product: UHCI Host Controller usb usb3: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb3: SerialNumber: 0000:00:1a.0 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected uhci_hcd 0000:00:1a.1: PCI INT B -> GSI 21 (level, low) -> IRQ 21 uhci_hcd 0000:00:1a.1: UHCI Host Controller uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 4 uhci_hcd 0000:00:1a.1: irq 21, io base 0x000060a0 usb usb4: New USB device found, idVendor=1d6b, idProduct=0001 usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb4: Product: UHCI Host Controller usb usb4: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb4: SerialNumber: 0000:00:1a.1 usb usb4: configuration #1 chosen from 1 choice hub 4-0:1.0: USB hub found hub 4-0:1.0: 2 ports detected uhci_hcd 0000:00:1a.2: PCI INT D -> GSI 19 (level, low) -> IRQ 19 uhci_hcd 0000:00:1a.2: UHCI Host Controller uhci_hcd 0000:00:1a.2: new USB bus registered, assigned bus number 5 uhci_hcd 0000:00:1a.2: irq 19, io base 0x00006080 usb usb5: New USB device found, idVendor=1d6b, idProduct=0001 usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb5: Product: UHCI Host Controller usb usb5: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb5: SerialNumber: 0000:00:1a.2 usb usb5: configuration #1 chosen from 1 choice hub 5-0:1.0: USB hub found hub 5-0:1.0: 2 ports detected uhci_hcd 0000:00:1d.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 6 uhci_hcd 0000:00:1d.0: irq 23, io base 0x00006060 usb usb6: New USB device found, idVendor=1d6b, idProduct=0001 usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb6: Product: UHCI Host Controller usb usb6: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb6: SerialNumber: 0000:00:1d.0 usb usb6: configuration #1 chosen from 1 choice hub 6-0:1.0: USB hub found hub 6-0:1.0: 2 ports detected uhci_hcd 0000:00:1d.1: PCI INT B -> GSI 19 (level, low) -> IRQ 19 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 7 uhci_hcd 0000:00:1d.1: irq 19, io base 0x00006040 usb usb7: New USB device found, idVendor=1d6b, idProduct=0001 usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb7: Product: UHCI Host Controller usb usb7: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb7: SerialNumber: 0000:00:1d.1 usb usb7: configuration #1 chosen from 1 choice hub 7-0:1.0: USB hub found hub 7-0:1.0: 2 ports detected uhci_hcd 0000:00:1d.2: PCI INT C -> GSI 18 (level, low) -> IRQ 18 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 8 uhci_hcd 0000:00:1d.2: irq 18, io base 0x00006020 usb usb8: New USB device found, idVendor=1d6b, idProduct=0001 usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb8: Product: UHCI Host Controller usb usb8: Manufacturer: Linux 2.6.32-122.el6.x86_64 uhci_hcd usb usb8: SerialNumber: 0000:00:1d.2 usb usb8: configuration #1 chosen from 1 choice hub 8-0:1.0: USB hub found hub 8-0:1.0: 2 ports detected PNP: No PS/2 controller found. Probing ports directly. mice: PS/2 mouse device common for all mice rtc_cmos 00:03: RTC can wake from S4 rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs cpuidle: using governor ladder cpuidle: using governor menu usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 rtc_cmos 00:03: setting system clock to 2011-03-15 01:31:00 UTC (1300152660) Initalizing network drop monitor service Freeing unused kernel memory: 1228k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 1120k freed Freeing unused kernel memory: 1800k freed Mounting proc fiusb 4-1: new full speed USB device using uhci_hcd and address 2 lesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Free memory/Total memory (free %): 65560 / 108284 ( 60.5445 ) Loading jbd2.ko module Loading mbcache.ko module Loading ext4.ko module Loading crc-t10dif.ko module Loading sd_mod.ko module Loading ata_generic.ko module Loading dm-mod.ko module device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel Loading dm-log.ko module Loading dm-region-hash.ko module Loading dm-mirror.ko module Loading dm-zero.ko module Loading dm-snapshot.ko module usb 4-1: New USB device found, idVendor=046b, idProduct=ff10 usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3 usb 4-1: Product: Virtual Keyboard and Mouse usb 4-1: Manufacturer: American Megatrends Inc. usb 4-1: SerialNumber: serial Loading autofs4.ko module Loading sunrpc.ko module usb 4-1: configuration #1 chosen from 1 choice RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Loading freq_table.ko module input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.0/input/input3 Loading ipv6.ko module generic-usb 0003:046B:FF10.0001: input,hidraw0: USB HID v1.10 Keyboard [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1/input0 NET: Registered protocol family 10 lo: Disabled Privacy Extensions Loading microcode.ko module microcode: CPU0 sig=0x206e6, pf=0x4, revision=0x6 input: American Megatrends Inc. Virtual Keyboard and Mouse as /devices/pci0000:00/0000:00:1a.1/usb4/4-1/4-1:1.1/input/input4 platform microcode: firmware: requesting intel-ucode/06-2e-06 generic-usb 0003:046B:FF10.0002: input,hidraw1: USB HID v1.10 Mouse [American Megatrends Inc. Virtual Keyboard and Mouse] on usb-0000:00:1a.1-1/input1 usb 7-2: new low speed USB device using uhci_hcd and address 2 usb 7-2: New USB device found, idVendor=0624, idProduct=0307 usb 7-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0 usb 7-2: Product: Avocent DSRIQ-USB usb 7-2: Manufacturer: Avocent usb 7-2: configuration #1 chosen from 1 choice input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.0/input/input5 generic-usb 0003:0624:0307.0003: input,hidraw2: USB HID v1.10 Keyboard [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2/input0 input: Avocent Avocent DSRIQ-USB as /devices/pci0000:00/0000:00:1d.1/usb7/7-2/7-2:1.1/input/input6 generic-usb 0003:0624:0307.0004: input,hidraw3: USB HID v1.10 Mouse [Avocent Avocent DSRIQ-USB] on usb-0000:00:1d.1-2/input1 Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba Loading hed.ko module Loading i2c-core.ko module Loading iTCO_vendor_support.ko module iTCO_vendor_support: vendor-support=0 Loading edac_core.ko module EDAC MC: Ver: 2.1.0 Mar 9 2011 Loading sg.ko module Loading dca.ko module dca service started, version 1.12.1 Loading cdrom.ko module Loading pata_acpi.ko module pata_acpi 0000:00:1f.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19 pata_acpi 0000:00:1f.2: PCI INT A disabled pata_acpi 0000:00:1f.5: PCI INT D -> GSI 21 (level, low) -> IRQ 21 pata_acpi 0000:00:1f.5: PCI INT D disabled Loading ata_piix.ko module ata_piix 0000:00:1f.2: PCI INT A -> GSI 19 (level, low) -> IRQ 19 ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ] scsi0 : ata_piix scsi1 : ata_piix ata1: SATA max UDMA/133 cmd 0x6138 ctl 0x614c bmdma 0x6110 irq 19 ata2: SATA max UDMA/133 cmd 0x6130 ctl 0x6148 bmdma 0x6118 irq 19 ata_piix 0000:00:1f.5: PCI INT D -> GSI 21 (level, low) -> IRQ 21 ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ] scsi2 : ata_piix scsi3 : ata_piix ata3: SATA max UDMA/133 cmd 0x6128 ctl 0x6144 bmdma 0x60f0 irq 21 ata4: SATA max UDMA/133 cmd 0x6120 ctl 0x6140 bmdma 0x60f8 irq 21 ata4: SATA link down (SStatus 0 SControl 300) ata3: SATA link down (SStatus 4 SControl 300) ata2.00: SATA link down (SStatus 0 SControl 300) ata2.01: SATA link down (SStatus 0 SControl 300) ata1.00: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.01: SATA link down (SStatus 0 SControl 300) ata1.00: ATAPI: Optiarc DVD RW AD-7580S, FX04, max UDMA/100 ata1.00: configured for UDMA/100 scsi 0:0:0:0: CD-ROM Optiarc DVD RW AD-7580S FX04 PQ: 0 ANSI: 5 scsi 0:0:0:0: Attached scsi generic sg0 type 5 Loading megaraid_sas.ko module megasas: 00.00.05.29-RH1 Tue. Dec. 7 17:00:00 PDT 2010 megasas: 0x1000:0x0079:0x8086:0x9261: bus 7:slot 0:func 0 megaraid_sas 0000:07:00.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24 megasas: Waiting for FW to come to ready state megasas: FW now in Ready state megasas_init_mfi: fw_support_ieee=0 megasas: INIT adapter done megaraid_sas: fw state:c0000000 megasas: fwstate:c0000000, dis_OCR=0 DRHD: handling fault status reg 102 INTR-REMAP: Request device [[01:00.0] fault index 32 INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear INFO: task insmod:201 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 201 1 0x00000000 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038 Call Trace: [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20 [<ffffffff812806d1>] pci_device_probe+0x101/0x120 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90 [<ffffffff81339d8e>] driver_attach+0x1e/0x20 [<ffffffff81339690>] bus_add_driver+0x200/0x300 [<ffffffff8133a5c6>] driver_register+0x76/0x140 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b INFO: task insmod:201 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 201 1 0x00000000 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038 Call Trace: [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20 [<ffffffff812806d1>] pci_device_probe+0x101/0x120 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90 [<ffffffff81339d8e>] driver_attach+0x1e/0x20 [<ffffffff81339690>] bus_add_driver+0x200/0x300 [<ffffffff8133a5c6>] driver_register+0x76/0x140 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b INFO: task insmod:201 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 201 1 0x00000000 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038 Call Trace: [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20 [<ffffffff812806d1>] pci_device_probe+0x101/0x120 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90 [<ffffffff81339d8e>] driver_attach+0x1e/0x20 [<ffffffff81339690>] bus_add_driver+0x200/0x300 [<ffffffff8133a5c6>] driver_register+0x76/0x140 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b DRHD: handling fault status reg 202 INTR-REMAP: Request device [[01:00.0] fault index 33 INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear DRHD: handling fault status reg 302 INTR-REMAP: Request device [[01:00.0] fault index 35 INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear DRHD: handling fault status reg 402 INTR-REMAP: Request device [[01:00.0] fault index 31 INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear DRHD: handling fault status reg 502 INTR-REMAP: Request device [[01:00.0] fault index 37 INTR-REMAP:[fault reason 34] Present field in the IRTE entry is clear INFO: task insmod:201 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 201 1 0x00000000 ffff880006f6bbb8 0000000000000082 ffff880004015f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f6bb98 ffffffff8105c5cc ffff880006f4b038 ffff880006f6bfd8 000000000000f558 ffff880006f4b038 Call Trace: [<ffffffff8105c5cc>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108e1ee>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa01b7745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108df00>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa01bacc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa01b8500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa01c1b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f4e7>] local_pci_probe+0x17/0x20 [<ffffffff812806d1>] pci_device_probe+0x101/0x120 [<ffffffff81339e52>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339ff0>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a29b>] __driver_attach+0xab/0xb0 [<ffffffff8133a1f0>] ? __driver_attach+0x0/0xb0 [<ffffffff81339254>] bus_for_each_dev+0x64/0x90 [<ffffffff81339d8e>] driver_attach+0x1e/0x20 [<ffffffff81339690>] bus_add_driver+0x200/0x300 [<ffffffff8133a5c6>] driver_register+0x76/0x140 [<ffffffff81174e7b>] ? __register_chrdev+0x8b/0xf0 [<ffffffff81280936>] __pci_register_driver+0x56/0xd0 [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa01cc0a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac7ff>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b RHEL6.1-20110311.3 installed, system reserved for debug. *** Bug 688334 has been marked as a duplicate of this bug. *** Created attachment 485874 [details]
serial console log
Created attachment 485875 [details]
sosreport
------- Comment From masbock.com 2011-03-17 13:50 EDT------- From the call traces it looks like the driver is waiting for an interrupt in response to megasas_issue_blocked_cmd. Perhaps this is an issue with MSI-x initialization in the kdump kernel. Therefore it might be worthwhile to boot both kernels with megaraid_sas.disable_msix=1 to see if the problem goes away. (This is not meant as a workaround, only a debugging step.) ------- Comment From masbock.com 2011-03-17 20:24 EDT------- This is a snippet from the kernel messages: It looks like megaraid_sas allocated a legacy (non-msix) interrupt vector. I'd guess that the device still thinks it's using MSI-x The following is a vt-d error. It is the result of devices still using MSI vectors activate by the main kernel for which there are no interrupt remapping table entries. This one could have been caused by the MSI in response to the megaraid device to the megasas_issue_blocked_cmd. (But it could also belong to the network device)
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Cause
Loading megaraid_sas driver and hardware in kdump kernel.
Consequence
Kdump kernel is unable to process any further, and insmod command is blocked with those messages.
INFO: task insmod:201 blocked for more than 120 seconds.
Workaround
None found yet.
------- Comment From shubgoya.com 2011-03-18 08:25 EDT------- There is one more addition to the platform list affected by this issue. Below are the details:- x3650M3 - 01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 9240 (rev 03) Thx, Shubham ------- Comment From masbock.com 2011-03-18 14:27 EDT------- (In reply to comment #16) > From the call traces it looks like the driver is waiting for an interrupt in > response to megasas_issue_blocked_cmd. Perhaps this is an issue with MSI-x > initialization in the kdump kernel. > Therefore it might be worthwhile to boot both kernels with > megaraid_sas.disable_msix=1 to see if the problem goes away. (This is not meant > as a workaround, only a debugging step.) The boot parameter really is megaraid_sas.msix_disable=1 (I had it backwards). Shubham just verified that this works. This means that if the megaraid_sas driver doesn't use MSI-x the kdump kernel boots. As a next step we should back port the following patch: [PATCH 5/15] megaraid_sas: Fix probe_one to clear MSI-X flags in kdump http://marc.info/?l=linux-scsi&m=129816856305895&w=2 to see if that fixes the issues. ------- Comment From masbock.com 2011-03-21 13:21 EDT------- (In reply to comment #21) > (In reply to comment #20) > > (In reply to comment #16) > > As a next step we should back port the following patch: > > > > [PATCH 5/15] megaraid_sas: Fix probe_one to clear MSI-X flags in kdump > > http://marc.info/?l=linux-scsi&m=129816856305895&w=2 > > > > to see if that fixes the issues. > > > Is this patch accepted in mainline? if we want this bug fixed in 6.1, we need > to pack port it and test it asap. please give an estimate as when the pack port > patch will be ready. > I thought I had seen the patch last Friday on kernel.org in 2.6.39-git8. But I don't see it anymore in git10. Would it be possible to have somebody from LSI to comment on the status of this patch in main-line? We can provide a back port and test results for the patch quickly. ------- Comment From masbock.com 2011-03-21 13:54 EDT------- I just verified that the patch is indeed in the latest main-line kernel git tree. I can be found here: http://www.kernel.org/diff/diffview.cgi?file=/pub/linux/kernel/v2.6/snapshots/patch-2.6.38-git10.bz2
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
Diffed Contents:
@@ -1,9 +1,3 @@
-Cause
- Loading megaraid_sas driver and hardware in kdump kernel.
-Consequence
- Kdump kernel is unable to process any further, and insmod command is blocked with those messages.
+Loading the megaraid_sas driver on the kdump kernel will result in the insmod command being blocked, returning messages similar to:
-INFO: task insmod:201 blocked for more than 120 seconds.
+INFO: task insmod:201 blocked for more than 120 seconds.-
-Workaround
- None found yet.
Created attachment 486704 [details] Patch to reset msix in megaraid_sas driver in kdump kernel ------- Comment on attachment From masbock.com 2011-03-21 19:24 EDT------- This is a back port of a patch that resets msix in the megaraid_sas driver in the kdump kernel. The patch was originally posted by Adam Radford from LSI here: http://marc.info/?l=linux-scsi&m=129816856305895&w=2 "Back port" is really an exaggeration, it's only shifted by 4 lines. This patch depends on the "reset_devices" boot parameter being passed. This parameter is by default passed in the kdump kernel. Therefore no special documentation is needed. It tested this patch with the linux-2.6.32-122.el6.x86_64 kernel and obtained a vmcore. The megaraid_sas driver worked fine in the kdump kernel with the patch. This patches fixes the issue. ------- Comment From tpnoonan.com 2011-03-22 15:58 EDT------- red hat please consider as exception for rhel6.1, thnx Hello, This is requested as a blocker for 6.1. Thank You Joe Kachuck I posted the patch for review. This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux maintenance release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Update release for currently deployed products. This request is not yet committed for inclusion in an Update release. Deleted Technical Notes Contents. Old Contents: Loading the megaraid_sas driver on the kdump kernel will result in the insmod command being blocked, returning messages similar to: INFO: task insmod:201 blocked for more than 120 seconds. ------- Comment From ramesyr1.com 2011-03-29 00:07 EDT------- I tested the patch on Linux 2.6.32-122.el6.x86_64 kernel ,after triggering the kdump megaraid_sas driver is loading without any issues but vmcore is not generated.I could only see an empty directory under /var/crash with no vmcore in it. FYI, [root@mx3850x5 crash]# cd 127.0.0.1-2011-03-23-20:17:28 ;ll total 0 I also noticed from console log that while loading SELinux policy ,the makedumpfile process is getting killed by a oom-killer while it is trying to save the vmcore. from the above observation ,I disabled the SELinux policy and triggered the kdump but the system got hanged(console log for hang point attached), after a longtime i rebooted the machine manually and the interesting thing was it has generated the vmcore of zero size. FYI, ----SELinux disabled---- [root@mx3850x5 ~]# getenforce Permissive -----Zero size vmcore------- [root@mx3850x5 ~]# cd /var/crash/2011-03-23-21:44/ ; ll total 0 -rw-------. 1 root root 0 Mar 23 21:44 vmcore. Machine Type = X3850-X5 system memory = 256GB * complete serial console log is attached. Created attachment 488322 [details]
serial console log
------- Comment (attachment only) From ramesyr1.com 2011-03-29 00:11 EDT-------
(In reply to comment #25) > ------- Comment From ramesyr1.com 2011-03-29 00:07 EDT------- > I tested the patch on Linux 2.6.32-122.el6.x86_64 kernel ,after triggering the > kdump megaraid_sas driver is loading without any issues but vmcore is not > generated.I could only see an empty directory under /var/crash with no vmcore > in it. > system memory = 256GB This look like another issue, please try, before we open another bug report, to reserve a larger memory size for the kdump kernel. It can happen is some corner cases that default value isn't enough. Try some larger values, the kernel option is crashkernel, so for example crashkernel=256M@256M or higher. ------- Comment From ramesyr1.com 2011-03-29 08:35 EDT------- (In reply to comment #32) > > This look like another issue, please try, before we open another bug report, to > reserve a larger memory size for the kdump kernel. It can happen is some corner > cases that default value isn't enough. Try some larger values, the kernel > option is crashkernel, so for example crashkernel=256M@256M or higher. I tried with crashkernel=512M ,the issue goes away however the vmcore genearation fails with crashkernel=auto . But according to the kdump documentation at Documentation/kdump/kdump.txt from RHEL6, the crashkernel=auto reserves memory according to following table: Memory size Reserved memory =========== =============== [4G, 12G) 256M [12G, 128G) 512M [128G, 256G) 768M [256G, 378G) 1024M [378G, 512G) 1536M [512G, 768G) 2048M [768G, ) 3072M According to the above table 1024M should have been reserved for crashkernel when specified crashkernel=auto but it is reserving only 129M. Since the auto feature does not work as expected in kdump documentation, this looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a separate BUG for this issue? (In reply to comment #28) > Since the auto feature does not work as expected in kdump documentation, this > looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a > separate BUG for this issue? Yes please, open a new Bug for this issue. I'm seeing the same problem on RHEL 5.6 (In reply to comment #30) > I'm seeing the same problem on RHEL 5.6 Do you mean the msi-x issue or the problem with the size of the crashkernel? The msi-x issue. (In reply to comment #32) > The msi-x issue. Let us continue this RHEL5 issue here -> bz#692099#c5 ------- Comment From iranna.ankad.com 2011-04-06 09:39 EDT------- (In reply to comment #35) > (In reply to comment #28) > > Since the auto feature does not work as expected in kdump documentation, this > > looks to be BUG in crashkernel=auto feature. Shall I go ahead and raise a > > separate BUG for this issue? > > Yes please, open a new Bug for this issue. Hello Red Hat FYI, We have opened below bugzilla to track this new issue. Bug 71062 - RH692764 :crashkernel=auto feature does not work as per the kdump documentation Now, coming back to the original issue for which we are using this bugzilla (i.e kdump hang on megaraid), the patch which we have submitted/tested is still under your review. Please let us know further status on that. Thanks! (In reply to comment #34) > Hello Red Hat FYI, > We have opened below bugzilla to track this new issue. > > Bug 71062 - RH692764 :crashkernel=auto feature does not work as per the > kdump documentation I hope I added the right person to this bug. > Now, coming back to the original issue for which we are using this bugzilla > (i.e kdump hang on megaraid), > the patch which we have submitted/tested is still under your review. Please let > us know further status on that. The patch is in POSTED state, it will updated accordingly when the patch gets included. This fix is approved and planned for inclusion in snapshot 3. Patch(es) available on kernel-2.6.32-130.el6 ------- Comment From iranna.ankad.com 2011-04-08 06:43 EDT------- (In reply to comment #42) > This fix is approved and planned for inclusion in snapshot 3. > Patch(es) available on kernel-2.6.32-130.el6 Red Hat, Thanks for the updates. We will retest once snap3 is available. ------- Comment From ramesyr1.com 2011-04-14 07:46 EDT------- I verified the kdump on snap3 , kdump worked fine and generated the vmcore successfully. Thanks ! ------- Comment From iranna.ankad.com 2011-04-14 10:16 EDT------- (In reply to comment #46) > I verified the kdump on snap3 , kdump worked fine and generated the vmcore > successfully. > > Thanks ! Ramesh, Thanks for the snap3 results. Red Hat, We can close this bug now. Thanks! ------- Comment From iranna.ankad.com 2011-04-14 10:16 EDT------- Closing... An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on therefore solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2011-0542.html |
Description of problem: [root@dell-pet310-02 ~]# rpm -q kernel kexec-tools kernel-2.6.32-119.el6.x86_64 kexec-tools-2.0.0-167.el6.x86_64 [root@dell-pet310-02 ~]# touch /etc/kdump.conf [root@dell-pet310-02 ~]# service kdump restart Stopping kdump: [ OK ] Detected change(s) the following file(s): /etc/kdump.conf Rebuilding /boot/initrd-2.6.32-119.el6.x86_64kdump.img Your running kernel is using more than 70% of the amount of space you reserved for kdump, you should consider increasing your crashkerne[WARNING]tion Starting kdump: [ OK ] [root@dell-pet310-02 ~]# tail /etc/kdump.conf #core_collector cp --sparse=always #extra_bins /bin/cp #link_delay 60 #kdump_post /var/crash/scripts/kdump-post.sh #extra_bins /usr/bin/lftp #disk_timeout 30 #extra_modules gfs2 #options modulename options #default shell [root@dell-pet310-02 ~]# echo c > /proc/sysrq-trigger ----------------------------------------------------------- dell-pet310-02.lab.bos.redhat.com login: readahead-collector: starting delayed readahead-collector: sorting readahead-collector: finished bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney bfa 0000:04:00.0: Base port (WWN = 10:00:00:05:33:26:9e:74) lost fabric conney SysRq : Trigger a crash BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0x20 PGD 23220a067 PUD 234e40067 PMD 0 Oops: 0002 [#1] SMP last sysfs file: /sys/firmware/memmap/7/type CPU 4 Modules linked in: autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log power_] Modules linked in: autofs4 sunrpc ipv6 dm_mirror dm_region_hash dm_log power_] Pid: 5485, comm: bash Not tainted 2.6.32-119.el6.x86_64 #1 PowerEdge T310 RIP: 0010:[<ffffffff81318fe6>] [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0 RSP: 0018:ffff880232d7de18 EFLAGS: 00010096 RAX: 0000000000000010 RBX: 0000000000000063 RCX: 0000000000000c99 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063 RBP: ffff880232d7de18 R08: ffffffff81b9de20 R09: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000001 R12: 0000000000000000 R13: ffffffff81a99a40 R14: 0000000000000286 R15: 0000000000000007 FS: 00007f560c1c7700(0000) GS:ffff88002f680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000232dca000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process bash (pid: 5485, threadinfo ffff880232d7c000, task ffff8802318974c0) Stack: ffff880232d7de68 ffffffff813192a2 ffff8802318974c0 ffff880200000000 <0> 0000000d80df4018 0000000000000002 ffff8802342f7e00 00007f560c1cf000 <0> 0000000000000002 fffffffffffffffb ffff880232d7de98 ffffffff8131935e Call Trace: [<ffffffff813192a2>] __handle_sysrq+0x132/0x1a0 [<ffffffff8131935e>] write_sysrq_trigger+0x4e/0x50 [<ffffffff811d33fe>] proc_reg_write+0x7e/0xc0 [<ffffffff81170c58>] vfs_write+0xb8/0x1a0 [<ffffffff810d1652>] ? audit_syscall_entry+0x272/0x2a0 [<ffffffff81171691>] sys_write+0x51/0x90 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b Code: d0 88 81 23 81 f0 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 RIP [<ffffffff81318fe6>] sysrq_handle_crash+0x16/0x20 RSP <ffff880232d7de18> CR2: 0000000000000000 Initializing cgroup subsys cpuset Initializing cgroup subsys cpu Linux version 2.6.32-119.el6.x86_64 (mockbuild.bos.redhat.com) 1 Command line: ro root=/dev/mapper/vg_dellpet31002-lv_root rd_LVM_LV=vg_dellpeK KERNEL supported cpus: Intel GenuineIntel AMD AuthenticAMD Centaur CentaurHauls BIOS-provided physical RAM map: BIOS-e820: 0000000000000100 - 0000000000099000 (usable) BIOS-e820: 0000000000100000 - 00000000bf699000 (usable) BIOS-e820: 00000000bf699000 - 00000000bf6af000 (reserved) BIOS-e820: 00000000bf6af000 - 00000000bf6ce000 (ACPI data) BIOS-e820: 00000000bf6ce000 - 00000000c0000000 (reserved) BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved) BIOS-e820: 00000000fe000000 - 0000000100000000 (reserved) BIOS-e820: 0000000100000000 - 0000000240000000 (usable) last_pfn = 0x240000 max_arch_pfn = 0x400000000 user-defined physical RAM map: user: 0000000000000000 - 00000000000a0000 (usable) user: 00000000020a0000 - 0000000009ffb000 (usable) user: 00000000bf699000 - 00000000bf6af000 (reserved) user: 00000000bf6af000 - 00000000bf6ce000 (ACPI data) user: 00000000bf6ce000 - 00000000c0000000 (reserved) user: 00000000e0000000 - 00000000f0000000 (reserved) user: 00000000fe000000 - 0000000100000000 (reserved) DMI 2.6 present. last_pfn = 0x9ffb max_arch_pfn = 0x400000000 x86 PAT enabled: cpu 0, old 0x7010600070106, new 0x7010600070106 init_memory_mapping: 0000000000000000-0000000009ffb000 RAMDISK: 09a2e000 - 09fee12f ACPI: RSDP 00000000000f0a80 00024 (v02 DELL ) ACPI: XSDT 00000000000f0b80 00094 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: FACP 00000000bf6c3bb4 000F4 (v03 DELL PE_SC3 00000001 DELL 00000001) ACPI: DSDT 00000000bf6af000 03E43 (v01 DELL PE_SC3 00000001 INTL 20050624) ACPI: FACS 00000000bf6c6000 00040 ACPI: APIC 00000000bf6c3478 00152 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: SPCR 00000000bf6c35cc 00050 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: HPET 00000000bf6c3620 00038 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: DM__ 00000000bf6c365c 000A8 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: MCFG 00000000bf6c3850 0003C (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: WD__ 00000000bf6c3890 00134 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: SLIC 00000000bf6c39c8 00024 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: ERST 00000000bf6b2fc4 00270 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: HEST 00000000bf6b3234 003A8 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: BERT 00000000bf6b2e44 00030 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: EINJ 00000000bf6b2e74 00150 (v01 DELL PE_SC3 00000001 DELL 00000001) ACPI: TCPA 00000000bf6c3b4c 00064 (v02 DELL PE_SC3 00000001 DELL 00000001) ACPI: SSDT 00000000bf6c7000 02784 (v01 INTEL PPM RCM 80000001 INTL 20061109) No NUMA configuration found Faking a node at 0000000000000000-0000000009ffb000 Bootmem setup node 0 0000000000000000-0000000009ffb000 NODE_DATA [0000000000009000 - 000000000003cfff] bootmap [000000000003d000 - 000000000003e3ff] pages 2 (7 early reservations) ==> bootmem [0000000000 - 0009ffb000] #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] #2 [0003000000 - 0003f453a4] TEXT DATA BSS ==> [0003000000 - 0003f453a4] #3 [0009a2e000 - 0009fee12f] RAMDISK ==> [0009a2e000 - 0009fee12f] #4 [0000099800 - 0000100000] BIOS reserved ==> [0000099800 - 0000100000] #5 [0003f46000 - 0003f46459] BRK ==> [0003f46000 - 0003f46459] #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] found SMP MP-table at [ffff8800000fe710] fe710 Zone PFN ranges: DMA 0x00000001 -> 0x00001000 DMA32 0x00001000 -> 0x00100000 Normal 0x00100000 -> 0x00100000 Movable zone start PFN for each node early_node_map[2] active PFN ranges 0: 0x00000001 -> 0x000000a0 0: 0x000020a0 -> 0x00009ffb ACPI: PM-Timer IO Port: 0x808 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x04] enabled) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x06] enabled) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x01] enabled) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x03] enabled) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x05] enabled) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x28] disabled) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x29] disabled) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x2a] disabled) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x2b] disabled) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x2c] disabled) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x2d] disabled) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x2e] disabled) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x2f] disabled) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x30] disabled) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x31] disabled) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x32] disabled) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x33] disabled) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x34] disabled) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x35] disabled) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x36] disabled) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x37] disabled) ACPI: LAPIC (acpi_id[0x19] lapic_id[0x38] disabled) ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x39] disabled) ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x3a] disabled) ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x3b] disabled) ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x3c] disabled) ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x3d] disabled) ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x3e] disabled) ACPI: LAPIC (acpi_id[0x20] lapic_id[0x3f] disabled) ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) ACPI: IOAPIC (id[0x00] address[0xfec00000] gsi_base[0]) IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Using ACPI (MADT) for SMP configuration information ACPI: HPET id: 0x8086a701 base: 0xfed00000 SMP: Allowing 32 CPUs, 24 hotplug CPUs PM: Registered nosave memory: 00000000000a0000 - 00000000020a0000 Allocating PCI resources starting at 9ffb000 (gap: 9ffb000:b569e000) Booting paravirtualized kernel on bare hardware NR_CPUS:4096 nr_cpumask_bits:32 nr_cpu_ids:32 nr_node_ids:1 PERCPU: Embedded 30 pages/cpu @ffff880002a00000 s92440 r8192 d22248 u131072 pcpu-alloc: s92440 r8192 d22248 u131072 alloc=1*2097152 pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 pcpu-alloc: [0] 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Built 1 zonelists in Node order, mobility grouping on. Total pages: 32202 Policy zone: DMA32 Kernel command line: ro root=/dev/mapper/vg_dellpet31002-lv_root rd_LVM_LV=vgK Misrouted IRQ fixup and polling support enabled This may significantly impact system performance Disabling memory control group subsystem PID hash table entries: 512 (order: 0, 4096 bytes) Checking aperture... No AGP bridge found AMD-Vi disabled by default: pass amd_iommu=on to enable Memory: 101076k/163820k available (5005k kernel code, 32772k absent, 29972k r) Hierarchical RCU implementation. NR_IRQS:33024 nr_irqs:664 Extended CMOS year: 2000 do_IRQ: 0.105 No irq handler for vector (irq -1) Console: colour VGA+ 80x25 console [ttyS1] enabled HPET: 8 timers in total, 5 timers will be used for per-cpu timer Fast TSC calibration using PIT Detected 2926.091 MHz processor. Calibrating delay loop (skipped), value calculated using timer frequency.. 58) pid_max: default: 32768 minimum: 301 Security Framework initialized SELinux: Initializing. Dentry cache hash table entries: 16384 (order: 5, 131072 bytes) Inode-cache hash table entries: 8192 (order: 4, 65536 bytes) Mount-cache hash table entries: 256 Initializing cgroup subsys ns Initializing cgroup subsys cpuacct Initializing cgroup subsys memory Initializing cgroup subsys devices Initializing cgroup subsys freezer Initializing cgroup subsys net_cls Initializing cgroup subsys blkio CPU: Physical Processor ID: 0 CPU: Processor Core ID: 0 mce: CPU supports 9 MCE banks using mwait in idle threads. SMP alternatives: switching to UP code ACPI: Core revision 20090903 ftrace: converting mcount calls to 0f 1f 44 00 00 ftrace: allocating 20654 entries in 81 pages Setting APIC routing to physical flat ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 CPU0: Intel(R) Xeon(R) CPU X3470 @ 2.93GHz stepping 05 Performance Events: PEBS fmt1+, Nehalem events, Broken BIOS detected, using s. [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 186 is 53003c) NMI watchdog disabled for cpu0: unable to create perf event: -2 Brought up 1 CPUs Total of 1 processors activated (5852.18 BogoMIPS). devtmpfs: initialized regulator: core version 0.5 NET: Registered protocol family 16 ACPI FADT declares the system doesn't support PCIe ASPM, so disable it ACPI: bus type pci registered PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255 PCI: MCFG area at e0000000 reserved in E820 PCI: Using MMCONFIG at e0000000 - efffffff PCI: Using configuration type 1 for base access bio: create slab <bio-0> at 0 ACPI: BIOS _OSI(Linux) query ignored ACPI: Interpreter enabled ACPI: (supports S0 S4 S5) ACPI: Using IOAPIC for interrupt routing ACPI: No dock devices found. ACPI: PCI Root Bridge [PCI0] (0000:00) pci 0000:00:00.0: PME# supported from D0 D3hot D3cold pci 0000:00:00.0: PME# disabled pci 0000:00:03.0: PME# supported from D0 D3hot D3cold pci 0000:00:03.0: PME# disabled pci 0000:00:05.0: PME# supported from D0 D3hot D3cold pci 0000:00:05.0: PME# disabled pci 0000:00:1a.0: PME# supported from D0 D3hot D3cold pci 0000:00:1a.0: PME# disabled pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold pci 0000:00:1c.0: PME# disabled pci 0000:00:1c.2: PME# supported from D0 D3hot D3cold pci 0000:00:1c.2: PME# disabled pci 0000:00:1c.3: PME# supported from D0 D3hot D3cold pci 0000:00:1c.3: PME# disabled pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold pci 0000:00:1c.4: PME# disabled pci 0000:00:1d.0: PME# supported from D0 D3hot D3cold pci 0000:00:1d.0: PME# disabled pci 0000:04:00.0: PME# supported from D0 D3hot pci 0000:04:00.0: PME# disabled pci 0000:04:00.1: PME# supported from D0 D3hot pci 0000:04:00.1: PME# disabled pci 0000:02:00.0: PME# supported from D0 D3hot D3cold pci 0000:02:00.0: PME# disabled pci 0000:02:00.1: PME# supported from D0 D3hot D3cold pci 0000:02:00.1: PME# disabled pci 0000:06:00.0: PME# supported from D0 D3hot D3cold pci 0000:06:00.0: PME# disabled pci 0000:06:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it w' pci 0000:07:00.0: PME# supported from D0 D3hot D3cold pci 0000:07:00.0: PME# disabled pci 0000:07:00.0: disabling ASPM on pre-1.1 PCIe device. You can enable it w' pci 0000:05:00.0: PME# supported from D0 D3hot pci 0000:05:00.0: PME# disabled pci 0000:05:00.1: PME# supported from D0 D3hot pci 0000:05:00.1: PME# disabled pci 0000:00:1e.0: transparent bridge ACPI: PCI Interrupt Link [LK00] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK01] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK02] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK03] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK04] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK05] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK06] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. ACPI: PCI Interrupt Link [LK07] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled. HEST: HEST table parsing is initialized. vgaarb: device added: PCI:0000:01:03.0,decodes=io+mem,owns=io+mem,locks=none vgaarb: loaded SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb PCI: Using ACPI for IRQ routing NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default hpet0: at MMIO 0xfed00000, IRQs 2, 8, 24, 25, 26, 27, 28, 0 hpet0: 8 comparators, 64-bit 14.318180 MHz counter Switching to clocksource tsc pnp: PnP ACPI init ACPI: bus type pnp registered pnp: PnP ACPI: found 13 devices ACPI: ACPI bus type pnp unregistered system 00:07: ioport range 0x800-0x87f has been reserved system 00:07: ioport range 0x880-0x8ff has been reserved system 00:07: ioport range 0x900-0x91f has been reserved system 00:07: ioport range 0x920-0x923 has been reserved system 00:07: ioport range 0x924-0x924 has been reserved system 00:07: ioport range 0xca0-0xca7 has been reserved system 00:07: ioport range 0xca9-0xcab has been reserved system 00:07: ioport range 0xcad-0xcaf has been reserved system 00:08: ioport range 0xca8-0xca8 has been reserved system 00:08: ioport range 0xcac-0xcac has been reserved system 00:0a: iomem range 0xe0000000-0xefffffff has been reserved system 00:0c: iomem range 0xfed50000-0xfed51fff has been reserved pci 0000:04:00.1: BAR 6: address space collision on of device [0xdf100000-0xd] pci 0000:00:03.0: PCI bridge to [bus 04-04] pci 0000:00:03.0: PCI bridge, secondary bus 0000:04 pci 0000:00:03.0: bridge window [io disabled] pci 0000:00:03.0: bridge window [0xdf100000-0xdf2fffff] pci 0000:00:03.0: bridge window [0xc0000000-0xc00fffff] pci 0000:00:05.0: PCI bridge to [bus 03-03] pci 0000:00:05.0: PCI bridge, secondary bus 0000:03 pci 0000:00:05.0: bridge window [0xf000-0xffff] pci 0000:00:05.0: bridge window [0xdf300000-0xdf3fffff] pci 0000:00:05.0: bridge window [mem pref disabled] pci 0000:00:1c.0: PCI bridge to [bus 02-02] pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02 pci 0000:00:1c.0: bridge window [io disabled] pci 0000:00:1c.0: bridge window [0xda000000-0xddffffff] pci 0000:00:1c.0: bridge window [mem pref disabled] pci 0000:00:1c.2: PCI bridge to [bus 06-06] pci 0000:00:1c.2: PCI bridge, secondary bus 0000:06 pci 0000:00:1c.2: bridge window [0xe000-0xefff] pci 0000:00:1c.2: bridge window [0xdf400000-0xdf4fffff] pci 0000:00:1c.2: bridge window [mem pref disabled] pci 0000:00:1c.3: PCI bridge to [bus 07-07] pci 0000:00:1c.3: PCI bridge, secondary bus 0000:07 pci 0000:00:1c.3: bridge window [0xd000-0xdfff] pci 0000:00:1c.3: bridge window [0xdf500000-0xdf5fffff] pci 0000:00:1c.3: bridge window [mem pref disabled] pci 0000:00:1c.4: PCI bridge to [bus 05-05] pci 0000:00:1c.4: PCI bridge, secondary bus 0000:05 pci 0000:00:1c.4: bridge window [io disabled] pci 0000:00:1c.4: bridge window [mem disabled] pci 0000:00:1c.4: bridge window [0xd7000000-0xd97fffff] pci 0000:00:1e.0: PCI bridge to [bus 01-01] pci 0000:00:1e.0: PCI bridge, secondary bus 0000:01 pci 0000:00:1e.0: bridge window [io disabled] pci 0000:00:1e.0: bridge window [0xde000000-0xdeffffff] pci 0000:00:1e.0: bridge window [0xd9800000-0xd9ffffff] pci 0000:00:03.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 pci 0000:00:05.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 NET: Registered protocol family 2 IP route cache hash table entries: 1024 (order: 1, 8192 bytes) TCP established hash table entries: 4096 (order: 4, 65536 bytes) TCP bind hash table entries: 4096 (order: 4, 65536 bytes) TCP: Hash tables configured (established 4096 bind 4096) TCP reno registered NET: Registered protocol family 1 Trying to unpack rootfs image as initramfs... Freeing initrd memory: 5888k freed audit: initializing netlink socket (disabled) type=2000 audit(1299204427.598:1): initialized HugeTLB registered 2 MB page size, pre-allocated 0 pages VFS: Disk quotas dquot_6.5.2 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) msgmni has been set to 208 alg: No test for stdrng (krng) ksign: Installing public key data Loading keyring - Added public key 8CDF1AE31685CBAC - User ID: Red Hat, Inc. (Kernel Module GPG key) - Added public key D4A26C9CCD09BEDA - User ID: Red Hat Enterprise Linux Driver Update Program <secalert> Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 pciehp: PCI Express Hot Plug Controller Driver version: 0.4 acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 pci-stub: invalid id string "" input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 ACPI: Power Button [PWRF] processor LNXCPU:00: registered as cooling_device0 processor LNXCPU:01: registered as cooling_device1 processor LNXCPU:02: registered as cooling_device2 processor LNXCPU:03: registered as cooling_device3 processor LNXCPU:04: registered as cooling_device4 processor LNXCPU:05: registered as cooling_device5 processor LNXCPU:06: registered as cooling_device6 processor LNXCPU:07: registered as cooling_device7 ERST: Error Record Serialization Table (ERST) support is initialized. Non-volatile memory driver v1.3 Linux agpgart interface v0.103 crash memory driver: version 1.1 Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A �serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A brd: module loaded loop: module loaded input: Macintosh mouse button emulation as /devices/virtual/input/input1 Fixed MDIO Bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci_hcd 0000:00:1a.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22 ehci_hcd 0000:00:1a.0: EHCI Host Controller ehci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:1a.0: debug port 2 ehci_hcd 0000:00:1a.0: irq 22, io mem 0xdf0fc000 ehci_hcd 0000:00:1a.0: USB 2.0 started, EHCI 1.00 usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: EHCI Host Controller usb usb1: Manufacturer: Linux 2.6.32-119.el6.x86_64 ehci_hcd usb usb1: SerialNumber: 0000:00:1a.0 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ehci_hcd 0000:00:1d.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22 ehci_hcd 0000:00:1d.0: EHCI Host Controller ehci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2 ehci_hcd 0000:00:1d.0: debug port 2 ehci_hcd 0000:00:1d.0: irq 22, io mem 0xdf0fe000 ehci_hcd 0000:00:1d.0: USB 2.0 started, EHCI 1.00 usb usb2: New USB device found, idVendor=1d6b, idProduct=0002 usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb2: Product: EHCI Host Controller usb usb2: Manufacturer: Linux 2.6.32-119.el6.x86_64 ehci_hcd usb usb2: SerialNumber: 0000:00:1d.0 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver uhci_hcd: USB Universal Host Controller Interface driver PNP: No PS/2 controller found. Probing ports directly. mice: PS/2 mouse device common for all mice rtc_cmos 00:04: RTC can wake from S4 rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0 rtc0: alarms up to one day, y3k, 242 bytes nvram, hpet irqs cpuidle: using governor ladder cpuidle: using governor menu usbcore: registered new interface driver hiddev usbcore: registered new interface driver usbhid usbhid: v2.6:USB HID core driver TCP cubic registered Initializing XFRM netlink socket NET: Registered protocol family 17 registered taskstats version 1 rtc_cmos 00:04: setting system clock to 2011-03-04 02:07:10 UTC (1299204430) Initalizing network drop monitor service Freeing unused kernel memory: 1228k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 1120k freed Freeing unused kernel memory: 1800k freed Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Free memory/Total memory (free %): 75028 / 111116 ( 67.5222 ) Loading jbd2.ko module Loading mbcache.ko module Loading ext4.ko module Loading crc-t10dif.ko module Loading sd_mod.ko module Loading ata_generic.ko module Loading dm-mod.ko module device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel Loading dm-log.ko module Loading dm-region-hash.ko module Loading dm-mirror.ko module Loading dm-zero.ko module Loading dm-snapshot.ko module Loading autofs4.ko module Loading sunrpc.ko module RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. Loading ipv6.ko module NET: Registered protocol family 10 lo: Disabled Privacy Extensions Loading hwmon.ko module Loading libcrc32c.ko module Loading mdio.ko module Loading e1000e.ko module e1000e: Intel(R) PRO/1000 Network Driver - 1.2.20-k2 e1000e: Copyright (c) 1999 - 2010 Intel Corporation. e1000e 0000:06:00.0: Disabling ASPM L1 e1000e 0000:06:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18 usb 1-1: new high speed USB device using ehci_hcd and address 2 e1000e 0000:06:00.0: eth0: (PCI Express:2.5GB/s:Width x1) 00:15:17:ef:95:29 e1000e 0000:06:00.0: eth0: Intel(R) PRO/1000 Network Connection e1000e 0000:06:00.0: eth0: MAC: 1, PHY: 4, PBA No: D28777-005 e1000e 0000:07:00.0: Disabling ASPM L1 e1000e 0000:07:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19 usb 1-1: New USB device found, idVendor=8087, idProduct=0020 usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 usb 1-1: configuration #1 chosen from 1 choice hub 1-1:1.0: USB hub found hub 1-1:1.0: 6 ports detected usb 2-1: new high speed USB device using ehci_hcd and address 2 e1000e 0000:07:00.0: eth1: (PCI Express:2.5GB/s:Width x1) 00:15:17:ef:10:e9 e1000e 0000:07:00.0: eth1: Intel(R) PRO/1000 Network Connection e1000e 0000:07:00.0: eth1: MAC: 1, PHY: 4, PBA No: D28777-005 Loading bnx2.ko module bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.17 (July 18, 20) bnx2 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 bnx2 0000:02:00.0: pci_enable_pcie_error_reporting failed 0xfffffffb bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-mips-09-6.0.17.fw bnx2 0000:02:00.0: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw bnx2 0000:02:00.0: eth2: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Ex2 bnx2 0000:02:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17 bnx2 0000:02:00.1: pci_enable_pcie_error_reporting failed 0xfffffffb bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-mips-09-6.0.17.fw bnx2 0000:02:00.1: firmware: requesting bnx2/bnx2-rv2p-09-6.0.17.fw bnx2 0000:02:00.1: eth3: Broadcom NetXtreme II BCM5716 1000Base-T (C0) PCI Ex3 Loading microcodusb 2-1: New USB device found, idVendor=8087, idProduct=0020 e.ko module usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 microcode: CPU0 sig=0x106e5, pf=0x2, revision=0x4 platform microcode: firmware: requesting intel-ucode/06-1e-05 usb 2-1: configuration #1 chosen from 1 choice hub 2-1:1.0: USB hub found hub 2-1:1.0: 8 ports detected usb 1-1.1: new high speed USB device using ehci_hcd and address 3 usb 1-1.1: New USB device found, idVendor=0424, idProduct=2514 usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 usb 1-1.1: configuration #1 chosen from 1 choice hub 1-1.1:1.0: USB hub found hub 1-1.1:1.0: 2 ports detected Microcode Update Driver: v2.00 <tigran.co.uk>, Peter Oruba Loading dcdbas.ko module dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2) Loading hed.ko module Loading iTCO_vendor_support.ko miTCO_vendor_support: vendor-support=0 odule Loading shpchp.ko module shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 Loading sg.ko module Loading bna.ko module Brocade 10G Ethernet driver bnad_pci_probe : (0xffff8800088f2000, 0xffffffffa02a7720) PCI Func : (1) bna 0000:04:00.1: firmware: requesting ctfw_cna.bin bna 0000:04:00.1: PCI INT A -> GSI 16 (level, low) -> IRQ 16 bar0 mapped to ffffc90010580000, len 262144 bna: GRO enabled, using kernel stack GRO Loading cdrom.ko module Loading pata_acpi.ko module pata_acpi 0000:00:1f.2: PCI INT A -> GSI 20 (level, low) -> IRQ 20 pata_acpi 0000:00:1f.2: PCI INT A disabled pata_acpi 0000:00:1f.5: PCI INT B -> GSI 21 (level, low) -> IRQ 21 pata_acpi 0000:00:1f.5: PCI INT B disabled Loading ata_piix.ko module ata_piix 0000:00:1f.2: PCI INT A -> GSI 20 (level, low) -> IRQ 20 ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ] scsi0 : ata_piix scsi1 : ata_piix ata1: SATA max UDMA/133 cmd 0xcca0 ctl 0xcc90 bmdma 0xccc0 irq 20 ata2: SATA max UDMA/133 cmd 0xcca8 ctl 0xcc94 bmdma 0xccc8 irq 20 ata_piix 0000:00:1f.5: PCI INT B -> GSI 21 (level, low) -> IRQ 21 ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ] scsi2 : ata_piix scsi3 : ata_piix ata3: SATA max UDMA/133 cmd 0xccb0 ctl 0xcc98 bmdma 0xcce0 irq 21 ata4: SATA max UDMA/133 cmd 0xccb8 ctl 0xcc9c bmdma 0xcce8 irq 21 ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata3.00: ATAPI: PLDS DVD-ROM DH-16D5S, VD15, max UDMA/100 ata3.00: configured for UDMA/100 ata2.00: failed to resume link (SControl 0) ata4: failed to resume link (SControl 0) ata4: SATA link down (SStatus 4 SControl 0) ata1.01: failed to resume link (SControl 0) ata1.00: SATA link down (SStatus 4 SControl 300) ata1.01: SATA link down (SStatus 4 SControl 0) ata2.01: failed to resume link (SControl 0) ata2.00: SATA link down (SStatus 4 SControl 0) ata2.01: SATA link down (SStatus 4 SControl 0) scsi 2:0:0:0: CD-ROM PLDS DVD-ROM DH-16D5S VD15 PQ: 0 ANSI: 5 scsi 2:0:0:0: Attached scsi generic sg0 type 5 Loading megaraid_sas.ko module megasas: 00.00.05.29-RH1 Tue. Dec. 7 17:00:00 PDT 2010 megasas: 0x1000:0x0079:0x1028:0x1f16: bus 3:slot 0:func 0 megaraid_sas 0000:03:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16 megasas: Waiting for FW to come to ready state megasas: FW now in Ready state megasas_init_mfi: fw_support_ieee=67108864 megasas: INIT adapter done megaraid_sas: fw state:c0000000 megasas: fwstate:c0000000, dis_OCR=0 INFO: task insmod:230 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 230 1 0x00000000 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8 Call Trace: [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f257>] local_pci_probe+0x17/0x20 [<ffffffff81280441>] pci_device_probe+0x101/0x120 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90 [<ffffffff81339afe>] driver_attach+0x1e/0x20 [<ffffffff81339400>] bus_add_driver+0x200/0x300 [<ffffffff8133a336>] driver_register+0x76/0x140 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b INFO: task insmod:230 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 230 1 0x00000000 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8 Call Trace: [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f257>] local_pci_probe+0x17/0x20 [<ffffffff81280441>] pci_device_probe+0x101/0x120 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90 [<ffffffff81339afe>] driver_attach+0x1e/0x20 [<ffffffff81339400>] bus_add_driver+0x200/0x300 [<ffffffff8133a336>] driver_register+0x76/0x140 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b INFO: task insmod:230 blocked for more than 120 seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. insmod D 0000000000000000 0 230 1 0x00000000 ffff880006f9dbb8 0000000000000082 ffff880002a15f40 0000000000000000 0000000000000000 0000000000000082 ffff880006f9db98 ffffffff8105c5ac ffff880006fa30f8 ffff880006f9dfd8 000000000000f558 ffff880006fa30f8 Call Trace: [<ffffffff8105c5ac>] ? try_to_wake_up+0xcc/0x400 [<ffffffff8108dfce>] ? prepare_to_wait+0x4e/0x80 [<ffffffffa02dd745>] megasas_issue_blocked_cmd+0x85/0xc0 [megaraid_sas] [<ffffffff8108dce0>] ? autoremove_wake_function+0x0/0x40 [<ffffffffa02e0cc3>] megasas_start_aen+0x143/0x2d0 [megaraid_sas] [<ffffffffa02de500>] ? megasas_isr+0x0/0x1f0 [megaraid_sas] [<ffffffffa02e7b13>] megasas_probe_one+0xc2d/0xee4 [megaraid_sas] [<ffffffff8127f257>] local_pci_probe+0x17/0x20 [<ffffffff81280441>] pci_device_probe+0x101/0x120 [<ffffffff81339bc2>] ? driver_sysfs_add+0x62/0x90 [<ffffffff81339d60>] driver_probe_device+0xa0/0x2a0 [<ffffffff8133a00b>] __driver_attach+0xab/0xb0 [<ffffffff81339f60>] ? __driver_attach+0x0/0xb0 [<ffffffff81338fc4>] bus_for_each_dev+0x64/0x90 [<ffffffff81339afe>] driver_attach+0x1e/0x20 [<ffffffff81339400>] bus_add_driver+0x200/0x300 [<ffffffff8133a336>] driver_register+0x76/0x140 [<ffffffff81174bfb>] ? __register_chrdev+0x8b/0xf0 [<ffffffff812806a6>] __pci_register_driver+0x56/0xd0 [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f2000>] ? megasas_init+0x0/0x1ea [megaraid_sas] [<ffffffffa02f20a1>] megasas_init+0xa1/0x1ea [megaraid_sas] [<ffffffff8100204c>] do_one_initcall+0x3c/0x1d0 [<ffffffff810ac5df>] sys_init_module+0xdf/0x250 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b Version-Release number of selected component (if applicable): kernel-2.6.32-119.el6.x86_64 kexec-tools-2.0.0-167.el6.x86_64 How reproducible: issue found on dell-pet310-02.lab.bos.redhat.com Steps to Reproduce: 1.Install RHEL6.1-20110224.2 2.Upgrade kernel/kexec-tools 3.Trigger crash Actual results: System got hang Expected results: vmcore saved Additional info: Another beaker test: https://beaker.engineering.redhat.com/recipes/119932