Bug 732128

Summary: kdump wrongly says 'No crashkernel parameter specified for running kernel' - requested crash kernel memory allocation silently fails
Product: Red Hat Enterprise Linux 5 Reporter: Charlie Brady <charlieb-fedora-bugzilla>
Component: kexec-toolsAssignee: Cong Wang <amwang>
Status: CLOSED ERRATA QA Contact: Xu Wang <xuwang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.6CC: amwang, mjrohl, qcai, rkhan, ruyang, xuwang
Target Milestone: rcKeywords: Documentation
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kexec-tools-1.102pre-157.el5 Doc Type: Bug Fix
Doc Text:
No documentation needed
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-08 04:08:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Charlie Brady 2011-08-19 21:48:00 UTC
I'm having trouble getting kdump configured and running, with CentOS 5 guest in vmware-server host.

kdump shows FAILED at startup, and I see a purported reason in /var/log/messages:

Aug 19 17:37:30 sdf kdump: No crashkernel parameter specified for running kernel
Aug 19 17:37:30 sdf kdump: failed to start up

But if I try to start kdump using:

sh -x /etc/init.d/kdump start

I can see that the diagnostic is bogus:

+ '[' -z '' ']'
++ cat /proc/cmdline
+ KDUMP_COMMANDLINE='ro root=/dev/main/root crashkernel=128M bootmem_debug=1'
++ uname -m
+ ARCH=i686
+ '[' i686 == ppc64 ']'
++ grep -v 00000000-00000000
++ grep 'Crash kernel' /proc/iomem
+ MEM_RESERVED=
+ '[' -z '' ']'
+ /usr/bin/logger -p info -t kdump 'No crashkernel parameter specified for running kernel'
+ return 1
+ '[' 1 '!=' 0 ']'
+ echo -n 'Starting kdump:'
Starting kdump:+ failure

The crashkernel parameter was in fact specified.

So there are two problems. First is simple - the diagnostic message is grossly misleading. Second problem (brobably more obscure) is that the Crash kernel memory is not being allocated in /proc/iomem. No reason for the failed allocation is logged in dmesg:

612MB HIGHMEM available.
896MB LOWMEM available.
bootmem::init_bootmem_core nid=0 start=0 map=85b end=38000 mapsize=7000
bootmem::reserve_bootmem addr=400000 size=462fff flags=0
bootmem::reserve_bootmem addr=0 size=1000 flags=0
bootmem::can_reserve_bootmem_core silent double reserve of PFN 0
bootmem::reserve_bootmem_core hm, page 00000000 reserved twice.
bootmem::reserve_bootmem addr=9f800 size=1000 flags=0
bootmem::can_reserve_bootmem_core silent double reserve of PFN 9f
bootmem::can_reserve_bootmem_core silent double reserve of PFN a0
bootmem::reserve_bootmem_core hm, page 0009f000 reserved twice.
bootmem::reserve_bootmem_core hm, page 000a0000 reserved twice.
bootmem::reserve_bootmem addr=1000 size=1000 flags=0
bootmem::can_reserve_bootmem_core silent double reserve of PFN 1
bootmem::reserve_bootmem_core hm, page 00001000 reserved twice.
bootmem::__alloc_bootmem_core size=1000 [1 pages] align=80 goal=0 limit=ffffffff
found SMP MP-table at 000f6ce0
bootmem::reserve_bootmem addr=f6ce0 size=1000 flags=0
bootmem::can_reserve_bootmem_core silent double reserve of PFN f6
bootmem::can_reserve_bootmem_core silent double reserve of PFN f7
bootmem::reserve_bootmem_core hm, page 000f6000 reserved twice.
bootmem::reserve_bootmem_core hm, page 000f7000 reserved twice.
bootmem::reserve_bootmem addr=9fd70 size=1000 flags=0
bootmem::can_reserve_bootmem_core silent double reserve of PFN 9f
bootmem::can_reserve_bootmem_core silent double reserve of PFN a0
bootmem::reserve_bootmem_core hm, page 0009f000 reserved twice.
bootmem::reserve_bootmem_core hm, page 000a0000 reserved twice.
bootmem::reserve_bootmem addr=37d8a000 size=2653f8 flags=0
Memory for crash kernel (0x0 to 0x0) notwithin permissible range
disabling kdump

[root@sdf ~]# rpm -q kexec-tools
kexec-tools-1.102pre-96.el5
[root@sdf ~]# rpm -q kernel
package kernel is not installed
[root@sdf ~]# rpm -q kernel-PAE
kernel-PAE-2.6.18-238.9.1.el5
[root@sdf ~]#

Comment 1 Charlie Brady 2011-08-19 22:01:43 UTC
> kernel-PAE-2.6.18-238.9.1.el5

I've verified that this same issue is seen with the non-PAE kernel - silent failure of crashkernel memory allocation. Google isn't finding me other reports of this problem - but I can't imagine I'm the only one seeing this.

Comment 2 Charlie Brady 2011-08-20 00:23:10 UTC
> + KDUMP_COMMANDLINE='ro root=/dev/main/root crashkernel=128M bootmem_debug=1'

OK, I've seen by looking at the kernel source arch/i386/kernel/setup.c that format for this version of kernel must include the @.

Perhaps this should be in the kernel source, as well as an appropriate fix to the message from kdump init script.


--- arch/i386/kernel/setup.c.orig       2011-08-19 20:18:27.861551000 -0400
+++ arch/i386/kernel/setup.c    2011-08-19 20:22:01.491770000 -0400
@@ -937,6 +937,10 @@
                                crashk_res.start = base;
                                crashk_res.end   = base + size - 1;
                        }
+                       else
+                       {
+                           printk(KERN_ERR "invalid crashkernel memory specification\n");
+                       }
                }
 #endif
 #ifdef CONFIG_PROC_VMCORE
@@ -1430,7 +1434,7 @@
                }
        }
        else {
-               printk(KERN_ERR "Memory for crash kernel (0x%lx to 0x%lx) not"
+               printk(KERN_ERR "Memory for crash kernel (0x%lx to 0x%lx) not "
                       "within permissible range\ndisabling kdump\n",
                       crashk_res.start, crashk_res.end);
                crashk_res.end = 0;

Comment 3 Cong Wang 2011-08-22 02:09:23 UTC
(In reply to comment #2)
Hi, Charlie

Right, we either need to put an error message in kernel or document this somewhere.

Thanks.

Comment 4 RHEL Program Management 2012-06-21 07:07:21 UTC
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux release.  Product Management has
requested further review of this request by Red Hat Engineering, for
potential inclusion in a Red Hat Enterprise Linux release for currently
deployed products.  This request is not yet committed for inclusion in
a release.

Comment 9 Michael Rohl 2012-12-10 23:46:21 UTC
I am running RHEL 5.8 and also have this problem.

Installed Packages
kexec-tools.x86_64                                     1.102pre-154.el5_8.1                                     installed

Here's what happens when I try to start it manually:

# sh -x /etc/init.d/kdump start
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' ']'
++ . /etc/profile.d/lang.sh
+++ sourced=0
+++ '[' -z '' -a -n en_US.UTF-8 ']'
+++ sourced=1
+++ '[' -n '' ']'
+++ '[' 1 = 1 ']'
+++ '[' -n en_US.UTF-8 ']'
+++ export LANG
+++ '[' -n '' ']'
+++ unset LC_ADDRESS
+++ '[' -n '' ']'
+++ unset LC_CTYPE
+++ '[' -n '' ']'
+++ unset LC_COLLATE
+++ '[' -n '' ']'
+++ unset LC_IDENTIFICATION
+++ '[' -n '' ']'
+++ unset LC_MEASUREMENT
+++ '[' -n '' ']'
+++ unset LC_MESSAGES
+++ '[' -n '' ']'
+++ unset LC_MONETARY
+++ '[' -n '' ']'
+++ unset LC_NAME
+++ '[' -n '' ']'
+++ unset LC_NUMERIC
+++ '[' -n '' ']'
+++ unset LC_PAPER
+++ '[' -n '' ']'
+++ unset LC_TELEPHONE
+++ '[' -n '' ']'
+++ unset LC_TIME
+++ '[' -n '' ']'
+++ unset LC_ALL
+++ '[' -n '' ']'
+++ unset LANGUAGE
+++ '[' -n '' ']'
+++ unset LINGUAS
+++ '[' -n '' ']'
+++ unset _XKB_CHARSET
+++ consoletype=pty
+++ '[' -z pty ']'
+++ '[' -n '' ']'
+++ '[' -n '' ']'
+++ '[' -n en_US.UTF-8 ']'
+++ case $LANG in
+++ '[' xterm = linux ']'
+++ unset SYSFONTACM SYSFONT
+++ unset sourced
+++ unset langfile
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ GRAPHICAL=yes
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ LOGLEVEL=3
+++ PROMPT=yes
+++ AUTOSWAP=no
++ '[' pty = serial ']'
++ '[' color '!=' verbose ']'
++ INITLOG_ARGS=-q
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
+ KEXEC=/sbin/kexec
+ BOOTDIR=/boot
+ KDUMP_KERNELVER=
+ KDUMP_INITRDEXT=
+ KDUMP_COMMANDLINE=
+ KDUMP_IDE_NOPROBE_COMMANDLINE=
+ KEXEC_ARGS=
+ KDUMP_CONFIG_FILE=/etc/kdump.conf
+ SSH_KEY_LOCATION=/root/.ssh/kdump_id_rsa
+ LOGGER='/usr/bin/logger -p info -t kdump'
++ uname -m
+ ARCH=x86_64
+ standard_kexec_args=-p
+ '[' -f /etc/sysconfig/kdump ']'
+ . /etc/sysconfig/kdump
++ KDUMP_KERNELVER=
++ KDUMP_COMMANDLINE=
++ KDUMP_COMMANDLINE_APPEND='irqpoll maxcpus=1 reset_devices'
++ KEXEC_ARGS=' --args-linux'
++ KDUMP_BOOTDIR=/boot
++ KDUMP_IMG=vmlinuz
++ KDUMP_IMG_EXT=
+ case "$1" in
+ '[' -s /proc/vmcore ']'
+ start
+ status
+ '[' '!' -e /sys/kernel/kexec_crash_loaded ']'
+ '[' -f /proc/xen/capabilities ']'
+ '[' -f /sys/hypervisor/type ']'
++ cat /sys/kernel/kexec_crash_loaded
+ rc=0
+ '[' 0 == 1 ']'
+ return 1
+ rc=1
+ '[' 1 == 2 ']'
+ '[' 1 == 0 ']'
+ check_config
+ '[' -z '' ']'
++ uname -r
+ local running_kernel=2.6.18-308.24.1.el5
++ echo 2.6.18-308.24.1.el5
++ sed -r 's/(smp|xen)//g'
+ kdump_kver=2.6.18-308.24.1.el5
+ '[' x86_64 == i686 ']'
+ kdump_kernel=/boot/vmlinuz-2.6.18-308.24.1.el5
+ kdump_initrd=/boot/initrd-2.6.18-308.24.1.el5kdump.img
+ '[' '!' -f /boot/vmlinuz-2.6.18-308.24.1.el5 ']'
+ '[' '!' -f /boot/initrd-2.6.18-308.24.1.el5kdump.img ']'
++ stat -c %Y /boot/initrd-2.6.18-308.24.1.el5kdump.img
+ image_time=1355166415
++ grep '^kdump_post' /etc/kdump.conf
++ cut '-d ' -f2
+ EXTRA_FILES=
++ grep '^kdump_pre' /etc/kdump.conf
++ cut '-d ' -f2
+ CHECK_FILE=
+ EXTRA_FILES=' '
++ grep '^extra_modules' /etc/kdump.conf
++ cut '-d ' -f2-
+ CHECK_FILE=
+ EXTRA_FILES='  '
++ grep '^extra_bins' /etc/kdump.conf
++ cut '-d ' -f2-
+ CHECK_FILE=
+ EXTRA_FILES='   '
++ grep '^extra_modules' /etc/kdump.conf
+ FORCE_REBUILD=
+ files='/etc/kdump.conf /boot/vmlinuz-2.6.18-308.24.1.el5    '
+ modified_files=
+ for file in '$files'
+ time_stamp=0
+ '[' -f /etc/kdump.conf ']'
++ stat -c %Y /etc/kdump.conf
+ time_stamp=1355166370
+ '[' 1355166370 -gt 1355166415 ']'
+ for file in '$files'
+ time_stamp=0
+ '[' -f /boot/vmlinuz-2.6.18-308.24.1.el5 ']'
++ stat -c %Y /boot/vmlinuz-2.6.18-308.24.1.el5
+ time_stamp=1354663247
+ '[' 1354663247 -gt 1355166415 ']'
+ '[' -n '' -a '!= ' ']'
+ '[' -n '' -a '!= ' ']'
+ return 0
+ '[' 0 '!=' 0 ']'
+ load_kdump
+ '[' -z '' ']'
++ cat /proc/cmdline
+ KDUMP_COMMANDLINE='ro root=/dev/VolGroup00/LogVol00 crashkernel="128M@16M"'
++ uname -m
+ ARCH=x86_64
+ '[' x86_64 == ppc64 ']'
++ grep 'Crash kernel' /proc/iomem
++ grep -v 00000000-00000000
+ MEM_RESERVED=
+ '[' -z '' ']'
+ /usr/bin/logger -p info -t kdump 'No crashkernel parameter specified for running kernel'
+ return 1
+ '[' 1 '!=' 0 ']'
+ echo -n 'Starting kdump:'
Starting kdump:+ failure
+ local rc=0
+ '[' color '!=' verbose -a -z '' ']'
+ echo_failure
+ '[' color = color ']'
+ echo -en '\033[60G'
                                                           + echo -n '['
[+ '[' color = color ']'
+ echo -en '\033[0;31m'
+ echo -n FAILED
FAILED+ '[' color = color ']'
+ echo -en '\033[0;39m'
+ echo -n ']'
]+ echo -ne '\r'
+ return 1
+ '[' -x /usr/bin/rhgb-client ']'
+ return 0
+ echo

+ /usr/bin/logger -p info -t kdump 'failed to start up'
+ return 1
+ exit 1




Thanks for any help!

Mike

Comment 10 errata-xmlrpc 2013-01-08 04:08:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0012.html