Bug 765696

Summary: Kdump is not working in RHEL 6.2
Product: Red Hat Enterprise Linux 6 Reporter: Linux engineering teams - Veritas <linux26port>
Component: kernelAssignee: Red Hat Kernel Manager <kernel-mgr>
Status: CLOSED NOTABUG QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-14 06:13:12 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
dmesg none

Description Linux engineering teams - Veritas 2011-12-09 06:28:38 UTC
Created attachment 544354 [details]
dmesg

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:It always reproducible


Steps to Reproduce:
1.install kernel 2.6.32-220.el6.x86_64 
2.if i type echo "c" > /proc/sysrq-trigger it crashes
3.But it wont take dump and error msgs are produced as shown in additional info
  
Actual results:it wont save dump and kernel panic - not syncing: Out of memory and no killable processes...


Expected results:dump is saved in /var/crash


Additional info:
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
PGD 22b003067 PUD 22bb71067 PMD 0 
Oops: 0002 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 1 
Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat xt_CHECKSUM iptable_mangle bridge stp llc autofs4 sunrpc cpufreq_ondemand powernow_k8 freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun kvm_amd kvm uinput bnx2 sg ses enclosure microcode dcdbas serio_raw e1000e k10temp amd64_edac_mod edac_core edac_mce_amd shpchp i2c_nforce2 ext4 mbcache jbd2 sr_mod cdrom usb_storage sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt pata_acpi ata_generic sata_nv megaraid_sas radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 2621, comm: bash Not tainted 2.6.32-220.el6.x86_64 #1 Dell Inc. PowerEdge R805/0F705T
RIP: 0010:[<ffffffff81325476>]  [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
RSP: 0018:ffff88012c593e18  EFLAGS: 00010096
RAX: 0000000000000010 RBX: 0000000000000063 RCX: 000000000000120d
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
RBP: ffff88012c593e18 R08: ffffffff81c00680 R09: 0000000000000000
R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff81afad40 R14: 0000000000000286 R15: 0000000000000004
FS:  00007f043d9e4700(0000) GS:ffff880133c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000022c3a9000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process bash (pid: 2621, threadinfo ffff88012c592000, task ffff880129664100)
Stack:
 ffff88012c593e68 ffffffff81325732 ffff880129664100 ffff880200000000
<0> 0000000d80df4018 0000000000000002 ffff88022c366800 00007f043d9f4000
<0> 0000000000000002 fffffffffffffffb ffff88012c593e98 ffffffff813257ee
Call Trace:
 [<ffffffff81325732>] __handle_sysrq+0x132/0x1a0
 [<ffffffff813257ee>] write_sysrq_trigger+0x4e/0x50
 [<ffffffff811dae8e>] proc_reg_write+0x7e/0xc0
 [<ffffffff811765d8>] vfs_write+0xb8/0x1a0
 [<ffffffff81176fe1>] sys_write+0x51/0x90
 [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
Code: d0 88 81 63 f4 fc 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 c7 05 5d 1e 77 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47 
RIP  [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
 RSP <ffff88012c593e18>
CR2: 0000000000000000
REWRITING MCP55 CFG REG
CFG = c1
do_IRQ: 0.178 No irq handler for vector (irq -1)
..MP-BIOS bug: 8254 timer not connected to IO-APIC
[Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR c0010000 is 530076)
ERST: Can not request iomem region <0xffff88000a193e60-0xffff100014327cc0> for ERST.
�Mounting proc filesystem
Mounting sysfs filesystem
Creating /dev
Creating initial device nodes
Free memory/Total memory (free %): 77900 / 113980 ( 68.3453 )
Loading freq_table.ko module
Loading mperf.ko module
Loading ipt_REJECT.ko module
Loading nf_defrag_ipv4.ko module
Loading ip_tables.ko module
Loading nf_conntrack.ko module
Loading ip6_tables.ko module
Loading ipv6.ko module
Loading macvlan.ko module
Loading tun.ko module
Loading kvm.ko module
Loading uinput.ko module
Loading bnx2.ko module
Loading sg.ko module
Loading enclosure.ko module
Loading microcode.ko module
 microcode: failed to load file amd-ucode/microcode_amd.bin
Loading dcdbas.ko module
Loading serio_raw.ko module
Loading e1000e.ko module
Loading k10temp.k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring disabled
ko module
k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
Loading edac_core.ko module
Loading edac_mce_amd.ko module
Loading shpchp.ko module
Loading mbcache.ko module
Loading jbd2.ko module
Loading cdrom.ko module
Loading usb-storage.ko module
Waiting 8 seconds for driver initialization.
Loading crc-t10dif.ko module
Loading scsi_tgt.ko module
Loading pata_acpi.ko module
Loading ata_generic.ko module
Loading sata_nv.ko module
Loading megaraid_sas.ko module
megasas: INIT adapter done
Loading i2c-core.ko module
Loading dm-mod.ko module
Loading cpufreq_ondemand.ko module
Loading powernow-k8.ko module
Loading nf_conntrack_ipv4.ko module
Loading iptable_filter.ko module
Loading ip6t_REJECT.ko module
Loading nf_defrag_ipv6.ko module
Loading xt_state.ko module
Loading ip6table_filter.ko module
Loading macvtap.ko module
Loading kvm-amd.ko module
Loading ses.ko module
Loading amd64_edac_mod.ko module
Loading i2c-nforce2.ko module
Loading ext4.ko module
Loading sr_mod.ko module
Loading sd_mod.ko module
Loading scsi_transport_fc.ko module
Loading drm.ko module
Loading i2c-algo-bit.ko module
Loading dm-log.ko module
Loading nf_conntrack_ipv6.ko module
Loading vhost_net.ko module
Loading qla2xxx.ko module
Loading ttm.ko module
Loading drm_kms_helper.ko module
Loading dm-region-hash.ko module
Loading radeon.ko module
Loading dm-mirror.ko module
Waiting for required block device discovery
Waiting for 4 sda-like device(s)...Found
Creating Block Devices
Creating block device loop0
Creating block device loop1
Creating block device loop2
Creating block device loop3
Creating block device loop4
Creating block device loop5
Creating block device loop6
Creating block device loop7
Creating block device ram0
Creating block device ram1
Creating block device ram10
Creating block device ram11
Creating block device ram12
Creating block device ram13
Creating block device ram14
Creating block device ram15
Creating block device ram2
Creating block device ram3
Creating block device ram4
Creating block device ram5
Creating block device ram6
Creating block device ram7
Creating block device ram8
Creating block device ram9
Creating block device sda
Creating block device sdb
Creating block device sdc
Creating block device sr0
Creating block device sr1
Saving to the local filesystem /dev/sda2
e2fsck 1.41.12 (17-May-2010)
fsck.ext4: No medium found while trying to open /dev/sda2

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

mount: mounting /dev/sda2 on /mnt failed: No medium found
Attempting to enter user-space to capture vmcore
Resetting kernel time value to BIOS time and timezone value to UTC.
Free memory/Total memory (free %): 63584 / 113980 ( 55.7852 )
Creating root device.
Free memory/Total memory (free %): 63476 / 113980 ( 55.6905 )
Checking root filesystem.
fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST)
fsck: fsck.auto: No such file or directory
Mounting root filesystem: mount -t ext4 /dev/sdb2 /sysroot
grep: /proc/cmdline: No such file or directory
Loading SELINUX policy
Free memory/Total memory (free %): 20412 / 113980 ( 17.9084 )
Switching to new root and running init.
                Welcome to Red Hat Enterprise Linux Server
Starting udev: Out of memory: Kill process 1397 (rc.sysinit) score 1 or sacrifice child
Killed process 1420, UID 0, (start_udev) total-vm:108524kB, anon-rss:516kB, file-rss:364kB
Out of memory: Kill process 1397 (rc.sysinit) score 1 or sacrifice child
Killed process 1397, UID 0, (rc.sysinit) total-vm:108524kB, anon-rss:472kB, file-rss:192kB
Out of memory: Kill process 1536 (udevadm) score 1 or sacrifice child
Killed process 1536, UID 0, (udevadm) total-vm:8448kB, anon-rss:92kB, file-rss:60kB
Kernel panic - not syncing: Out of memory and no killable processes...

Comment 2 Linux engineering teams - Veritas 2011-12-12 06:52:46 UTC
(In reply to comment #0)
> Created attachment 544354 [details]
> dmesg
> 
> Description of problem:
> 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:It always reproducible
> 
> 
> Steps to Reproduce:
> 1.install kernel 2.6.32-220.el6.x86_64 
> 2.if i type echo "c" > /proc/sysrq-trigger it crashes
> 3.But it wont take dump and error msgs are produced as shown in additional info
> 
> Actual results:it wont save dump and kernel panic - not syncing: Out of memory
> and no killable processes...
> 
> 
> Expected results:dump is saved in /var/crash
> 
> 
> Additional info:
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
> PGD 22b003067 PUD 22bb71067 PMD 0 
> Oops: 0002 [#1] SMP 
> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
> CPU 1 
> Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
> xt_CHECKSUM iptable_mangle bridge stp llc autofs4 sunrpc cpufreq_ondemand
> powernow_k8 freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
> iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun
> kvm_amd kvm uinput bnx2 sg ses enclosure microcode dcdbas serio_raw e1000e
> k10temp amd64_edac_mod edac_core edac_mce_amd shpchp i2c_nforce2 ext4 mbcache
> jbd2 sr_mod cdrom usb_storage sd_mod crc_t10dif qla2xxx scsi_transport_fc
> scsi_tgt pata_acpi ata_generic sata_nv megaraid_sas radeon ttm drm_kms_helper
> drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last
> unloaded: scsi_wait_scan]
> 
> Pid: 2621, comm: bash Not tainted 2.6.32-220.el6.x86_64 #1 Dell Inc. PowerEdge
> R805/0F705T
> RIP: 0010:[<ffffffff81325476>]  [<ffffffff81325476>]
> sysrq_handle_crash+0x16/0x20
> RSP: 0018:ffff88012c593e18  EFLAGS: 00010096
> RAX: 0000000000000010 RBX: 0000000000000063 RCX: 000000000000120d
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
> RBP: ffff88012c593e18 R08: ffffffff81c00680 R09: 0000000000000000
> R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000000
> R13: ffffffff81afad40 R14: 0000000000000286 R15: 0000000000000004
> FS:  00007f043d9e4700(0000) GS:ffff880133c00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 000000022c3a9000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process bash (pid: 2621, threadinfo ffff88012c592000, task ffff880129664100)
> Stack:
>  ffff88012c593e68 ffffffff81325732 ffff880129664100 ffff880200000000
> <0> 0000000d80df4018 0000000000000002 ffff88022c366800 00007f043d9f4000
> <0> 0000000000000002 fffffffffffffffb ffff88012c593e98 ffffffff813257ee
> Call Trace:
>  [<ffffffff81325732>] __handle_sysrq+0x132/0x1a0
>  [<ffffffff813257ee>] write_sysrq_trigger+0x4e/0x50
>  [<ffffffff811dae8e>] proc_reg_write+0x7e/0xc0
>  [<ffffffff811765d8>] vfs_write+0xb8/0x1a0
>  [<ffffffff81176fe1>] sys_write+0x51/0x90
>  [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Code: d0 88 81 63 f4 fc 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
> e5 0f 1f 44 00 00 c7 05 5d 1e 77 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00
> 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47 
> RIP  [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
>  RSP <ffff88012c593e18>
> CR2: 0000000000000000
> REWRITING MCP55 CFG REG
> CFG = c1
> do_IRQ: 0.178 No irq handler for vector (irq -1)
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR c0010000 is
> 530076)
> ERST: Can not request iomem region <0xffff88000a193e60-0xffff100014327cc0> for
> ERST.
> �Mounting proc filesystem
> Mounting sysfs filesystem
> Creating /dev
> Creating initial device nodes
> Free memory/Total memory (free %): 77900 / 113980 ( 68.3453 )
> Loading freq_table.ko module
> Loading mperf.ko module
> Loading ipt_REJECT.ko module
> Loading nf_defrag_ipv4.ko module
> Loading ip_tables.ko module
> Loading nf_conntrack.ko module
> Loading ip6_tables.ko module
> Loading ipv6.ko module
> Loading macvlan.ko module
> Loading tun.ko module
> Loading kvm.ko module
> Loading uinput.ko module
> Loading bnx2.ko module
> Loading sg.ko module
> Loading enclosure.ko module
> Loading microcode.ko module
>  microcode: failed to load file amd-ucode/microcode_amd.bin
> Loading dcdbas.ko module
> Loading serio_raw.ko module
> Loading e1000e.ko module
> Loading k10temp.k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring
> disabled
> ko module
> k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
> Loading edac_core.ko module
> Loading edac_mce_amd.ko module
> Loading shpchp.ko module
> Loading mbcache.ko module
> Loading jbd2.ko module
> Loading cdrom.ko module
> Loading usb-storage.ko module
> Waiting 8 seconds for driver initialization.
> Loading crc-t10dif.ko module
> Loading scsi_tgt.ko module
> Loading pata_acpi.ko module
> Loading ata_generic.ko module
> Loading sata_nv.ko module
> Loading megaraid_sas.ko module
> megasas: INIT adapter done
> Loading i2c-core.ko module
> Loading dm-mod.ko module
> Loading cpufreq_ondemand.ko module
> Loading powernow-k8.ko module
> Loading nf_conntrack_ipv4.ko module
> Loading iptable_filter.ko module
> Loading ip6t_REJECT.ko module
> Loading nf_defrag_ipv6.ko module
> Loading xt_state.ko module
> Loading ip6table_filter.ko module
> Loading macvtap.ko module
> Loading kvm-amd.ko module
> Loading ses.ko module
> Loading amd64_edac_mod.ko module
> Loading i2c-nforce2.ko module
> Loading ext4.ko module
> Loading sr_mod.ko module
> Loading sd_mod.ko module
> Loading scsi_transport_fc.ko module
> Loading drm.ko module
> Loading i2c-algo-bit.ko module
> Loading dm-log.ko module
> Loading nf_conntrack_ipv6.ko module
> Loading vhost_net.ko module
> Loading qla2xxx.ko module
> Loading ttm.ko module
> Loading drm_kms_helper.ko module
> Loading dm-region-hash.ko module
> Loading radeon.ko module
> Loading dm-mirror.ko module
> Waiting for required block device discovery
> Waiting for 4 sda-like device(s)...Found
> Creating Block Devices
> Creating block device loop0
> Creating block device loop1
> Creating block device loop2
> Creating block device loop3
> Creating block device loop4
> Creating block device loop5
> Creating block device loop6
> Creating block device loop7
> Creating block device ram0
> Creating block device ram1
> Creating block device ram10
> Creating block device ram11
> Creating block device ram12
> Creating block device ram13
> Creating block device ram14
> Creating block device ram15
> Creating block device ram2
> Creating block device ram3
> Creating block device ram4
> Creating block device ram5
> Creating block device ram6
> Creating block device ram7
> Creating block device ram8
> Creating block device ram9
> Creating block device sda
> Creating block device sdb
> Creating block device sdc
> Creating block device sr0
> Creating block device sr1
> Saving to the local filesystem /dev/sda2
> e2fsck 1.41.12 (17-May-2010)
> fsck.ext4: No medium found while trying to open /dev/sda2
> 
> The superblock could not be read or does not describe a correct ext2
> filesystem.  If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
> 
> mount: mounting /dev/sda2 on /mnt failed: No medium found
> Attempting to enter user-space to capture vmcore
> Resetting kernel time value to BIOS time and timezone value to UTC.
> Free memory/Total memory (free %): 63584 / 113980 ( 55.7852 )
> Creating root device.
> Free memory/Total memory (free %): 63476 / 113980 ( 55.6905 )
> Checking root filesystem.
> fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST)
> fsck: fsck.auto: No such file or directory
> Mounting root filesystem: mount -t ext4 /dev/sdb2 /sysroot
> grep: /proc/cmdline: No such file or directory
> Loading SELINUX policy
> Free memory/Total memory (free %): 20412 / 113980 ( 17.9084 )
> Switching to new root and running init.
>                 Welcome to Red Hat Enterprise Linux Server
> Starting udev: Out of memory: Kill process 1397 (rc.sysinit) score 1 or
> sacrifice child
> Killed process 1420, UID 0, (start_udev) total-vm:108524kB, anon-rss:516kB,
> file-rss:364kB
> Out of memory: Kill process 1397 (rc.sysinit) score 1 or sacrifice child
> Killed process 1397, UID 0, (rc.sysinit) total-vm:108524kB, anon-rss:472kB,
> file-rss:192kB
> Out of memory: Kill process 1536 (udevadm) score 1 or sacrifice child
> Killed process 1536, UID 0, (udevadm) total-vm:8448kB, anon-rss:92kB,
> file-rss:60kB
> Kernel panic - not syncing: Out of memory and no killable processes...

(In reply to comment #0)
> Created attachment 544354 [details]
> dmesg
> 
> Description of problem:
> 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:It always reproducible
> 
> 
> Steps to Reproduce:
> 1.install kernel 2.6.32-220.el6.x86_64 
> 2.if i type echo "c" > /proc/sysrq-trigger it crashes
> 3.But it wont take dump and error msgs are produced as shown in additional info
> 
> Actual results:it wont save dump and kernel panic - not syncing: Out of memory
> and no killable processes...
> 
> 
> Expected results:dump is saved in /var/crash
> 
> 
> Additional info:
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
> PGD 22b003067 PUD 22bb71067 PMD 0 
> Oops: 0002 [#1] SMP 
> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
> CPU 1 
> Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
> xt_CHECKSUM iptable_mangle bridge stp llc autofs4 sunrpc cpufreq_ondemand
> powernow_k8 freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
> iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun
> kvm_amd kvm uinput bnx2 sg ses enclosure microcode dcdbas serio_raw e1000e
> k10temp amd64_edac_mod edac_core edac_mce_amd shpchp i2c_nforce2 ext4 mbcache
> jbd2 sr_mod cdrom usb_storage sd_mod crc_t10dif qla2xxx scsi_transport_fc
> scsi_tgt pata_acpi ata_generic sata_nv megaraid_sas radeon ttm drm_kms_helper
> drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last
> unloaded: scsi_wait_scan]
> 
> Pid: 2621, comm: bash Not tainted 2.6.32-220.el6.x86_64 #1 Dell Inc. PowerEdge
> R805/0F705T
> RIP: 0010:[<ffffffff81325476>]  [<ffffffff81325476>]
> sysrq_handle_crash+0x16/0x20
> RSP: 0018:ffff88012c593e18  EFLAGS: 00010096
> RAX: 0000000000000010 RBX: 0000000000000063 RCX: 000000000000120d
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
> RBP: ffff88012c593e18 R08: ffffffff81c00680 R09: 0000000000000000
> R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000000
> R13: ffffffff81afad40 R14: 0000000000000286 R15: 0000000000000004
> FS:  00007f043d9e4700(0000) GS:ffff880133c00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 000000022c3a9000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process bash (pid: 2621, threadinfo ffff88012c592000, task ffff880129664100)
> Stack:
>  ffff88012c593e68 ffffffff81325732 ffff880129664100 ffff880200000000
> <0> 0000000d80df4018 0000000000000002 ffff88022c366800 00007f043d9f4000
> <0> 0000000000000002 fffffffffffffffb ffff88012c593e98 ffffffff813257ee
> Call Trace:
>  [<ffffffff81325732>] __handle_sysrq+0x132/0x1a0
>  [<ffffffff813257ee>] write_sysrq_trigger+0x4e/0x50
>  [<ffffffff811dae8e>] proc_reg_write+0x7e/0xc0
>  [<ffffffff811765d8>] vfs_write+0xb8/0x1a0
>  [<ffffffff81176fe1>] sys_write+0x51/0x90
>  [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Code: d0 88 81 63 f4 fc 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
> e5 0f 1f 44 00 00 c7 05 5d 1e 77 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00
> 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47 
> RIP  [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
>  RSP <ffff88012c593e18>
> CR2: 0000000000000000
> REWRITING MCP55 CFG REG
> CFG = c1
> do_IRQ: 0.178 No irq handler for vector (irq -1)
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR c0010000 is
> 530076)
> ERST: Can not request iomem region <0xffff88000a193e60-0xffff100014327cc0> for
> ERST.
> �Mounting proc filesystem
> Mounting sysfs filesystem
> Creating /dev
> Creating initial device nodes
> Free memory/Total memory (free %): 77900 / 113980 ( 68.3453 )
> Loading freq_table.ko module
> Loading mperf.ko module
> Loading ipt_REJECT.ko module
> Loading nf_defrag_ipv4.ko module
> Loading ip_tables.ko module
> Loading nf_conntrack.ko module
> Loading ip6_tables.ko module
> Loading ipv6.ko module
> Loading macvlan.ko module
> Loading tun.ko module
> Loading kvm.ko module
> Loading uinput.ko module
> Loading bnx2.ko module
> Loading sg.ko module
> Loading enclosure.ko module
> Loading microcode.ko module
>  microcode: failed to load file amd-ucode/microcode_amd.bin
> Loading dcdbas.ko module
> Loading serio_raw.ko module
> Loading e1000e.ko module
> Loading k10temp.k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring
> disabled
> ko module
> k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
> Loading edac_core.ko module
> Loading edac_mce_amd.ko module
> Loading shpchp.ko module
> Loading mbcache.ko module
> Loading jbd2.ko module
> Loading cdrom.ko module
> Loading usb-storage.ko module
> Waiting 8 seconds for driver initialization.
> Loading crc-t10dif.ko module
> Loading scsi_tgt.ko module
> Loading pata_acpi.ko module
> Loading ata_generic.ko module
> Loading sata_nv.ko module
> Loading megaraid_sas.ko module
> megasas: INIT adapter done
> Loading i2c-core.ko module
> Loading dm-mod.ko module
> Loading cpufreq_ondemand.ko module
> Loading powernow-k8.ko module
> Loading nf_conntrack_ipv4.ko module
> Loading iptable_filter.ko module
> Loading ip6t_REJECT.ko module
> Loading nf_defrag_ipv6.ko module
> Loading xt_state.ko module
> Loading ip6table_filter.ko module
> Loading macvtap.ko module
> Loading kvm-amd.ko module
> Loading ses.ko module
> Loading amd64_edac_mod.ko module
> Loading i2c-nforce2.ko module
> Loading ext4.ko module
> Loading sr_mod.ko module
> Loading sd_mod.ko module
> Loading scsi_transport_fc.ko module
> Loading drm.ko module
> Loading i2c-algo-bit.ko module
> Loading dm-log.ko module
> Loading nf_conntrack_ipv6.ko module
> Loading vhost_net.ko module
> Loading qla2xxx.ko module
> Loading ttm.ko module
> Loading drm_kms_helper.ko module
> Loading dm-region-hash.ko module
> Loading radeon.ko module
> Loading dm-mirror.ko module
> Waiting for required block device discovery
> Waiting for 4 sda-like device(s)...Found
> Creating Block Devices
> Creating block device loop0
> Creating block device loop1
> Creating block device loop2
> Creating block device loop3
> Creating block device loop4
> Creating block device loop5
> Creating block device loop6
> Creating block device loop7
> Creating block device ram0
> Creating block device ram1
> Creating block device ram10
> Creating block device ram11
> Creating block device ram12
> Creating block device ram13
> Creating block device ram14
> Creating block device ram15
> Creating block device ram2
> Creating block device ram3
> Creating block device ram4
> Creating block device ram5
> Creating block device ram6
> Creating block device ram7
> Creating block device ram8
> Creating block device ram9
> Creating block device sda
> Creating block device sdb
> Creating block device sdc
> Creating block device sr0
> Creating block device sr1
> Saving to the local filesystem /dev/sda2
> e2fsck 1.41.12 (17-May-2010)
> fsck.ext4: No medium found while trying to open /dev/sda2
> 
> The superblock could not be read or does not describe a correct ext2
> filesystem.  If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
> 
> mount: mounting /dev/sda2 on /mnt failed: No medium found
> Attempting to enter user-space to capture vmcore
> Resetting kernel time value to BIOS time and timezone value to UTC.
> Free memory/Total memory (free %): 63584 / 113980 ( 55.7852 )
> Creating root device.
> Free memory/Total memory (free %): 63476 / 113980 ( 55.6905 )
> Checking root filesystem.
> fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST)
> fsck: fsck.auto: No such file or directory
> Mounting root filesystem: mount -t ext4 /dev/sdb2 /sysroot
> grep: /proc/cmdline: No such file or directory
> Loading SELINUX policy
> Free memory/Total memory (free %): 20412 / 113980 ( 17.9084 )
> Switching to new root and running init.
>                 Welcome to Red Hat Enterprise Linux Server
> Starting udev: Out of memory: Kill process 1397 (rc.sysinit) score 1 or
> sacrifice child
> Killed process 1420, UID 0, (start_udev) total-vm:108524kB, anon-rss:516kB,
> file-rss:364kB
> Out of memory: Kill process 1397 (rc.sysinit) score 1 or sacrifice child
> Killed process 1397, UID 0, (rc.sysinit) total-vm:108524kB, anon-rss:472kB,
> file-rss:192kB
> Out of memory: Kill process 1536 (udevadm) score 1 or sacrifice child
> Killed process 1536, UID 0, (udevadm) total-vm:8448kB, anon-rss:92kB,
> file-rss:60kB
> Kernel panic - not syncing: Out of memory and no killable processes...
Updating Description of problem and version Release number

Description of problem:In RHEL 6.2 if we crash the kernel It wont save dump and
kernel is in panic state.Error messages as shown in additional information


Version-Release number of selected component (if applicable):RHEL 6.2 


How reproducible:It always reproducible


Steps to Reproduce: 
1.if we run command  echo "c" > /proc/sysrq-trigger it crashes the kernel
2.But it wont save dump and error messages are produced as shown in additional information

Actual results:Dump is not saved in /var/crash Directory


Expected results:Dump is saved in /var/crash Directory

Comment 3 Linux engineering teams - Veritas 2011-12-14 06:13:12 UTC
(In reply to comment #0)
> Created attachment 544354 [details]
> dmesg
> 
> Description of problem:
> 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:It always reproducible
> 
> 
> Steps to Reproduce:
> 1.install kernel 2.6.32-220.el6.x86_64 
> 2.if i type echo "c" > /proc/sysrq-trigger it crashes
> 3.But it wont take dump and error msgs are produced as shown in additional info
> 
> Actual results:it wont save dump and kernel panic - not syncing: Out of memory
> and no killable processes...
> 
> 
> Expected results:dump is saved in /var/crash
> 
> 
> Additional info:
> BUG: unable to handle kernel NULL pointer dereference at (null)
> IP: [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
> PGD 22b003067 PUD 22bb71067 PMD 0 
> Oops: 0002 [#1] SMP 
> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
> CPU 1 
> Modules linked in: ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat
> xt_CHECKSUM iptable_mangle bridge stp llc autofs4 sunrpc cpufreq_ondemand
> powernow_k8 freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4
> iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state
> nf_conntrack ip6table_filter ip6_tables ipv6 vhost_net macvtap macvlan tun
> kvm_amd kvm uinput bnx2 sg ses enclosure microcode dcdbas serio_raw e1000e
> k10temp amd64_edac_mod edac_core edac_mce_amd shpchp i2c_nforce2 ext4 mbcache
> jbd2 sr_mod cdrom usb_storage sd_mod crc_t10dif qla2xxx scsi_transport_fc
> scsi_tgt pata_acpi ata_generic sata_nv megaraid_sas radeon ttm drm_kms_helper
> drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last
> unloaded: scsi_wait_scan]
> 
> Pid: 2621, comm: bash Not tainted 2.6.32-220.el6.x86_64 #1 Dell Inc. PowerEdge
> R805/0F705T
> RIP: 0010:[<ffffffff81325476>]  [<ffffffff81325476>]
> sysrq_handle_crash+0x16/0x20
> RSP: 0018:ffff88012c593e18  EFLAGS: 00010096
> RAX: 0000000000000010 RBX: 0000000000000063 RCX: 000000000000120d
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000063
> RBP: ffff88012c593e18 R08: ffffffff81c00680 R09: 0000000000000000
> R10: 00000000ffffffff R11: 0000000000000000 R12: 0000000000000000
> R13: ffffffff81afad40 R14: 0000000000000286 R15: 0000000000000004
> FS:  00007f043d9e4700(0000) GS:ffff880133c00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 000000022c3a9000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process bash (pid: 2621, threadinfo ffff88012c592000, task ffff880129664100)
> Stack:
>  ffff88012c593e68 ffffffff81325732 ffff880129664100 ffff880200000000
> <0> 0000000d80df4018 0000000000000002 ffff88022c366800 00007f043d9f4000
> <0> 0000000000000002 fffffffffffffffb ffff88012c593e98 ffffffff813257ee
> Call Trace:
>  [<ffffffff81325732>] __handle_sysrq+0x132/0x1a0
>  [<ffffffff813257ee>] write_sysrq_trigger+0x4e/0x50
>  [<ffffffff811dae8e>] proc_reg_write+0x7e/0xc0
>  [<ffffffff811765d8>] vfs_write+0xb8/0x1a0
>  [<ffffffff81176fe1>] sys_write+0x51/0x90
>  [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
> Code: d0 88 81 63 f4 fc 81 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89
> e5 0f 1f 44 00 00 c7 05 5d 1e 77 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00
> 01 c9 c3 55 48 89 e5 0f 1f 44 00 00 8d 47 
> RIP  [<ffffffff81325476>] sysrq_handle_crash+0x16/0x20
>  RSP <ffff88012c593e18>
> CR2: 0000000000000000
> REWRITING MCP55 CFG REG
> CFG = c1
> do_IRQ: 0.178 No irq handler for vector (irq -1)
> ..MP-BIOS bug: 8254 timer not connected to IO-APIC
> [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR c0010000 is
> 530076)
> ERST: Can not request iomem region <0xffff88000a193e60-0xffff100014327cc0> for
> ERST.
> �Mounting proc filesystem
> Mounting sysfs filesystem
> Creating /dev
> Creating initial device nodes
> Free memory/Total memory (free %): 77900 / 113980 ( 68.3453 )
> Loading freq_table.ko module
> Loading mperf.ko module
> Loading ipt_REJECT.ko module
> Loading nf_defrag_ipv4.ko module
> Loading ip_tables.ko module
> Loading nf_conntrack.ko module
> Loading ip6_tables.ko module
> Loading ipv6.ko module
> Loading macvlan.ko module
> Loading tun.ko module
> Loading kvm.ko module
> Loading uinput.ko module
> Loading bnx2.ko module
> Loading sg.ko module
> Loading enclosure.ko module
> Loading microcode.ko module
>  microcode: failed to load file amd-ucode/microcode_amd.bin
> Loading dcdbas.ko module
> Loading serio_raw.ko module
> Loading e1000e.ko module
> Loading k10temp.k10temp 0000:00:18.3: unreliable CPU thermal sensor; monitoring
> disabled
> ko module
> k10temp 0000:00:19.3: unreliable CPU thermal sensor; monitoring disabled
> Loading edac_core.ko module
> Loading edac_mce_amd.ko module
> Loading shpchp.ko module
> Loading mbcache.ko module
> Loading jbd2.ko module
> Loading cdrom.ko module
> Loading usb-storage.ko module
> Waiting 8 seconds for driver initialization.
> Loading crc-t10dif.ko module
> Loading scsi_tgt.ko module
> Loading pata_acpi.ko module
> Loading ata_generic.ko module
> Loading sata_nv.ko module
> Loading megaraid_sas.ko module
> megasas: INIT adapter done
> Loading i2c-core.ko module
> Loading dm-mod.ko module
> Loading cpufreq_ondemand.ko module
> Loading powernow-k8.ko module
> Loading nf_conntrack_ipv4.ko module
> Loading iptable_filter.ko module
> Loading ip6t_REJECT.ko module
> Loading nf_defrag_ipv6.ko module
> Loading xt_state.ko module
> Loading ip6table_filter.ko module
> Loading macvtap.ko module
> Loading kvm-amd.ko module
> Loading ses.ko module
> Loading amd64_edac_mod.ko module
> Loading i2c-nforce2.ko module
> Loading ext4.ko module
> Loading sr_mod.ko module
> Loading sd_mod.ko module
> Loading scsi_transport_fc.ko module
> Loading drm.ko module
> Loading i2c-algo-bit.ko module
> Loading dm-log.ko module
> Loading nf_conntrack_ipv6.ko module
> Loading vhost_net.ko module
> Loading qla2xxx.ko module
> Loading ttm.ko module
> Loading drm_kms_helper.ko module
> Loading dm-region-hash.ko module
> Loading radeon.ko module
> Loading dm-mirror.ko module
> Waiting for required block device discovery
> Waiting for 4 sda-like device(s)...Found
> Creating Block Devices
> Creating block device loop0
> Creating block device loop1
> Creating block device loop2
> Creating block device loop3
> Creating block device loop4
> Creating block device loop5
> Creating block device loop6
> Creating block device loop7
> Creating block device ram0
> Creating block device ram1
> Creating block device ram10
> Creating block device ram11
> Creating block device ram12
> Creating block device ram13
> Creating block device ram14
> Creating block device ram15
> Creating block device ram2
> Creating block device ram3
> Creating block device ram4
> Creating block device ram5
> Creating block device ram6
> Creating block device ram7
> Creating block device ram8
> Creating block device ram9
> Creating block device sda
> Creating block device sdb
> Creating block device sdc
> Creating block device sr0
> Creating block device sr1
> Saving to the local filesystem /dev/sda2
> e2fsck 1.41.12 (17-May-2010)
> fsck.ext4: No medium found while trying to open /dev/sda2
> 
> The superblock could not be read or does not describe a correct ext2
> filesystem.  If the device is valid and it really contains an ext2
> filesystem (and not swap or ufs or something else), then the superblock
> is corrupt, and you might try running e2fsck with an alternate superblock:
>     e2fsck -b 8193 <device>
> 
> mount: mounting /dev/sda2 on /mnt failed: No medium found
> Attempting to enter user-space to capture vmcore
> Resetting kernel time value to BIOS time and timezone value to UTC.
> Free memory/Total memory (free %): 63584 / 113980 ( 55.7852 )
> Creating root device.
> Free memory/Total memory (free %): 63476 / 113980 ( 55.6905 )
> Checking root filesystem.
> fsck (busybox 1.15.1, 2010-11-30 08:10:31 EST)
> fsck: fsck.auto: No such file or directory
> Mounting root filesystem: mount -t ext4 /dev/sdb2 /sysroot
> grep: /proc/cmdline: No such file or directory
> Loading SELINUX policy
> Free memory/Total memory (free %): 20412 / 113980 ( 17.9084 )
> Switching to new root and running init.
>                 Welcome to Red Hat Enterprise Linux Server
> Starting udev: Out of memory: Kill process 1397 (rc.sysinit) score 1 or
> sacrifice child
> Killed process 1420, UID 0, (start_udev) total-vm:108524kB, anon-rss:516kB,
> file-rss:364kB
> Out of memory: Kill process 1397 (rc.sysinit) score 1 or sacrifice child
> Killed process 1397, UID 0, (rc.sysinit) total-vm:108524kB, anon-rss:472kB,
> file-rss:192kB
> Out of memory: Kill process 1536 (udevadm) score 1 or sacrifice child
> Killed process 1536, UID 0, (udevadm) total-vm:8448kB, anon-rss:92kB,
> file-rss:60kB
> Kernel panic - not syncing: Out of memory and no killable processes...

This is Configuration issue.closing the Bug.