I'm having trouble getting kdump configured and running, with CentOS 5 guest in vmware-server host. kdump shows FAILED at startup, and I see a purported reason in /var/log/messages: Aug 19 17:37:30 sdf kdump: No crashkernel parameter specified for running kernel Aug 19 17:37:30 sdf kdump: failed to start up But if I try to start kdump using: sh -x /etc/init.d/kdump start I can see that the diagnostic is bogus: + '[' -z '' ']' ++ cat /proc/cmdline + KDUMP_COMMANDLINE='ro root=/dev/main/root crashkernel=128M bootmem_debug=1' ++ uname -m + ARCH=i686 + '[' i686 == ppc64 ']' ++ grep -v 00000000-00000000 ++ grep 'Crash kernel' /proc/iomem + MEM_RESERVED= + '[' -z '' ']' + /usr/bin/logger -p info -t kdump 'No crashkernel parameter specified for running kernel' + return 1 + '[' 1 '!=' 0 ']' + echo -n 'Starting kdump:' Starting kdump:+ failure The crashkernel parameter was in fact specified. So there are two problems. First is simple - the diagnostic message is grossly misleading. Second problem (brobably more obscure) is that the Crash kernel memory is not being allocated in /proc/iomem. No reason for the failed allocation is logged in dmesg: 612MB HIGHMEM available. 896MB LOWMEM available. bootmem::init_bootmem_core nid=0 start=0 map=85b end=38000 mapsize=7000 bootmem::reserve_bootmem addr=400000 size=462fff flags=0 bootmem::reserve_bootmem addr=0 size=1000 flags=0 bootmem::can_reserve_bootmem_core silent double reserve of PFN 0 bootmem::reserve_bootmem_core hm, page 00000000 reserved twice. bootmem::reserve_bootmem addr=9f800 size=1000 flags=0 bootmem::can_reserve_bootmem_core silent double reserve of PFN 9f bootmem::can_reserve_bootmem_core silent double reserve of PFN a0 bootmem::reserve_bootmem_core hm, page 0009f000 reserved twice. bootmem::reserve_bootmem_core hm, page 000a0000 reserved twice. bootmem::reserve_bootmem addr=1000 size=1000 flags=0 bootmem::can_reserve_bootmem_core silent double reserve of PFN 1 bootmem::reserve_bootmem_core hm, page 00001000 reserved twice. bootmem::__alloc_bootmem_core size=1000 [1 pages] align=80 goal=0 limit=ffffffff found SMP MP-table at 000f6ce0 bootmem::reserve_bootmem addr=f6ce0 size=1000 flags=0 bootmem::can_reserve_bootmem_core silent double reserve of PFN f6 bootmem::can_reserve_bootmem_core silent double reserve of PFN f7 bootmem::reserve_bootmem_core hm, page 000f6000 reserved twice. bootmem::reserve_bootmem_core hm, page 000f7000 reserved twice. bootmem::reserve_bootmem addr=9fd70 size=1000 flags=0 bootmem::can_reserve_bootmem_core silent double reserve of PFN 9f bootmem::can_reserve_bootmem_core silent double reserve of PFN a0 bootmem::reserve_bootmem_core hm, page 0009f000 reserved twice. bootmem::reserve_bootmem_core hm, page 000a0000 reserved twice. bootmem::reserve_bootmem addr=37d8a000 size=2653f8 flags=0 Memory for crash kernel (0x0 to 0x0) notwithin permissible range disabling kdump [root@sdf ~]# rpm -q kexec-tools kexec-tools-1.102pre-96.el5 [root@sdf ~]# rpm -q kernel package kernel is not installed [root@sdf ~]# rpm -q kernel-PAE kernel-PAE-2.6.18-238.9.1.el5 [root@sdf ~]#
> kernel-PAE-2.6.18-238.9.1.el5 I've verified that this same issue is seen with the non-PAE kernel - silent failure of crashkernel memory allocation. Google isn't finding me other reports of this problem - but I can't imagine I'm the only one seeing this.
> + KDUMP_COMMANDLINE='ro root=/dev/main/root crashkernel=128M bootmem_debug=1' OK, I've seen by looking at the kernel source arch/i386/kernel/setup.c that format for this version of kernel must include the @. Perhaps this should be in the kernel source, as well as an appropriate fix to the message from kdump init script. --- arch/i386/kernel/setup.c.orig 2011-08-19 20:18:27.861551000 -0400 +++ arch/i386/kernel/setup.c 2011-08-19 20:22:01.491770000 -0400 @@ -937,6 +937,10 @@ crashk_res.start = base; crashk_res.end = base + size - 1; } + else + { + printk(KERN_ERR "invalid crashkernel memory specification\n"); + } } #endif #ifdef CONFIG_PROC_VMCORE @@ -1430,7 +1434,7 @@ } } else { - printk(KERN_ERR "Memory for crash kernel (0x%lx to 0x%lx) not" + printk(KERN_ERR "Memory for crash kernel (0x%lx to 0x%lx) not " "within permissible range\ndisabling kdump\n", crashk_res.start, crashk_res.end); crashk_res.end = 0;
(In reply to comment #2) Hi, Charlie Right, we either need to put an error message in kernel or document this somewhere. Thanks.
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux release for currently deployed products. This request is not yet committed for inclusion in a release.
I am running RHEL 5.8 and also have this problem. Installed Packages kexec-tools.x86_64 1.102pre-154.el5_8.1 installed Here's what happens when I try to start it manually: # sh -x /etc/init.d/kdump start + . /etc/init.d/functions ++ TEXTDOMAIN=initscripts ++ umask 022 ++ PATH=/sbin:/usr/sbin:/bin:/usr/bin ++ export PATH ++ '[' -z '' ']' ++ COLUMNS=80 ++ '[' -z '' ']' +++ /sbin/consoletype ++ CONSOLETYPE=pty ++ '[' -f /etc/sysconfig/i18n -a -z '' ']' ++ . /etc/profile.d/lang.sh +++ sourced=0 +++ '[' -z '' -a -n en_US.UTF-8 ']' +++ sourced=1 +++ '[' -n '' ']' +++ '[' 1 = 1 ']' +++ '[' -n en_US.UTF-8 ']' +++ export LANG +++ '[' -n '' ']' +++ unset LC_ADDRESS +++ '[' -n '' ']' +++ unset LC_CTYPE +++ '[' -n '' ']' +++ unset LC_COLLATE +++ '[' -n '' ']' +++ unset LC_IDENTIFICATION +++ '[' -n '' ']' +++ unset LC_MEASUREMENT +++ '[' -n '' ']' +++ unset LC_MESSAGES +++ '[' -n '' ']' +++ unset LC_MONETARY +++ '[' -n '' ']' +++ unset LC_NAME +++ '[' -n '' ']' +++ unset LC_NUMERIC +++ '[' -n '' ']' +++ unset LC_PAPER +++ '[' -n '' ']' +++ unset LC_TELEPHONE +++ '[' -n '' ']' +++ unset LC_TIME +++ '[' -n '' ']' +++ unset LC_ALL +++ '[' -n '' ']' +++ unset LANGUAGE +++ '[' -n '' ']' +++ unset LINGUAS +++ '[' -n '' ']' +++ unset _XKB_CHARSET +++ consoletype=pty +++ '[' -z pty ']' +++ '[' -n '' ']' +++ '[' -n '' ']' +++ '[' -n en_US.UTF-8 ']' +++ case $LANG in +++ '[' xterm = linux ']' +++ unset SYSFONTACM SYSFONT +++ unset sourced +++ unset langfile ++ '[' -z '' ']' ++ '[' -f /etc/sysconfig/init ']' ++ . /etc/sysconfig/init +++ BOOTUP=color +++ GRAPHICAL=yes +++ RES_COL=60 +++ MOVE_TO_COL='echo -en \033[60G' +++ SETCOLOR_SUCCESS='echo -en \033[0;32m' +++ SETCOLOR_FAILURE='echo -en \033[0;31m' +++ SETCOLOR_WARNING='echo -en \033[0;33m' +++ SETCOLOR_NORMAL='echo -en \033[0;39m' +++ LOGLEVEL=3 +++ PROMPT=yes +++ AUTOSWAP=no ++ '[' pty = serial ']' ++ '[' color '!=' verbose ']' ++ INITLOG_ARGS=-q ++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d' + KEXEC=/sbin/kexec + BOOTDIR=/boot + KDUMP_KERNELVER= + KDUMP_INITRDEXT= + KDUMP_COMMANDLINE= + KDUMP_IDE_NOPROBE_COMMANDLINE= + KEXEC_ARGS= + KDUMP_CONFIG_FILE=/etc/kdump.conf + SSH_KEY_LOCATION=/root/.ssh/kdump_id_rsa + LOGGER='/usr/bin/logger -p info -t kdump' ++ uname -m + ARCH=x86_64 + standard_kexec_args=-p + '[' -f /etc/sysconfig/kdump ']' + . /etc/sysconfig/kdump ++ KDUMP_KERNELVER= ++ KDUMP_COMMANDLINE= ++ KDUMP_COMMANDLINE_APPEND='irqpoll maxcpus=1 reset_devices' ++ KEXEC_ARGS=' --args-linux' ++ KDUMP_BOOTDIR=/boot ++ KDUMP_IMG=vmlinuz ++ KDUMP_IMG_EXT= + case "$1" in + '[' -s /proc/vmcore ']' + start + status + '[' '!' -e /sys/kernel/kexec_crash_loaded ']' + '[' -f /proc/xen/capabilities ']' + '[' -f /sys/hypervisor/type ']' ++ cat /sys/kernel/kexec_crash_loaded + rc=0 + '[' 0 == 1 ']' + return 1 + rc=1 + '[' 1 == 2 ']' + '[' 1 == 0 ']' + check_config + '[' -z '' ']' ++ uname -r + local running_kernel=2.6.18-308.24.1.el5 ++ echo 2.6.18-308.24.1.el5 ++ sed -r 's/(smp|xen)//g' + kdump_kver=2.6.18-308.24.1.el5 + '[' x86_64 == i686 ']' + kdump_kernel=/boot/vmlinuz-2.6.18-308.24.1.el5 + kdump_initrd=/boot/initrd-2.6.18-308.24.1.el5kdump.img + '[' '!' -f /boot/vmlinuz-2.6.18-308.24.1.el5 ']' + '[' '!' -f /boot/initrd-2.6.18-308.24.1.el5kdump.img ']' ++ stat -c %Y /boot/initrd-2.6.18-308.24.1.el5kdump.img + image_time=1355166415 ++ grep '^kdump_post' /etc/kdump.conf ++ cut '-d ' -f2 + EXTRA_FILES= ++ grep '^kdump_pre' /etc/kdump.conf ++ cut '-d ' -f2 + CHECK_FILE= + EXTRA_FILES=' ' ++ grep '^extra_modules' /etc/kdump.conf ++ cut '-d ' -f2- + CHECK_FILE= + EXTRA_FILES=' ' ++ grep '^extra_bins' /etc/kdump.conf ++ cut '-d ' -f2- + CHECK_FILE= + EXTRA_FILES=' ' ++ grep '^extra_modules' /etc/kdump.conf + FORCE_REBUILD= + files='/etc/kdump.conf /boot/vmlinuz-2.6.18-308.24.1.el5 ' + modified_files= + for file in '$files' + time_stamp=0 + '[' -f /etc/kdump.conf ']' ++ stat -c %Y /etc/kdump.conf + time_stamp=1355166370 + '[' 1355166370 -gt 1355166415 ']' + for file in '$files' + time_stamp=0 + '[' -f /boot/vmlinuz-2.6.18-308.24.1.el5 ']' ++ stat -c %Y /boot/vmlinuz-2.6.18-308.24.1.el5 + time_stamp=1354663247 + '[' 1354663247 -gt 1355166415 ']' + '[' -n '' -a '!= ' ']' + '[' -n '' -a '!= ' ']' + return 0 + '[' 0 '!=' 0 ']' + load_kdump + '[' -z '' ']' ++ cat /proc/cmdline + KDUMP_COMMANDLINE='ro root=/dev/VolGroup00/LogVol00 crashkernel="128M@16M"' ++ uname -m + ARCH=x86_64 + '[' x86_64 == ppc64 ']' ++ grep 'Crash kernel' /proc/iomem ++ grep -v 00000000-00000000 + MEM_RESERVED= + '[' -z '' ']' + /usr/bin/logger -p info -t kdump 'No crashkernel parameter specified for running kernel' + return 1 + '[' 1 '!=' 0 ']' + echo -n 'Starting kdump:' Starting kdump:+ failure + local rc=0 + '[' color '!=' verbose -a -z '' ']' + echo_failure + '[' color = color ']' + echo -en '\033[60G' + echo -n '[' [+ '[' color = color ']' + echo -en '\033[0;31m' + echo -n FAILED FAILED+ '[' color = color ']' + echo -en '\033[0;39m' + echo -n ']' ]+ echo -ne '\r' + return 1 + '[' -x /usr/bin/rhgb-client ']' + return 0 + echo + /usr/bin/logger -p info -t kdump 'failed to start up' + return 1 + exit 1 Thanks for any help! Mike
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2013-0012.html