Bug 517529 - kdump fails to start when capture kernel memory is reserved from 16M
Summary: kdump fails to start when capture kernel memory is reserved from 16M
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise MRG
Classification: Red Hat
Component: realtime-kernel
Version: 1.2
Hardware: x86_64
OS: All
low
medium
Target Milestone: ---
: ---
Assignee: Clark Williams
QA Contact: David Sommerseth
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-08-14 13:33 UTC by IBM Bug Proxy
Modified: 2016-05-22 23:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-09-15 09:05:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
IBM Linux Technology Center 55403 0 None None None Never

Description IBM Bug Proxy 2009-08-14 13:33:13 UTC
=Comment: #0=================================================
GOWRISHANKAR MUTHUKRISHNAN <gowrishankar.m.com> - 
Problem Description:

Kdump service fails to start in 2.6.31-rc5.2.el5rt when capture kernel memory is
reserved after 16M i.e crashkernel=128M@16M is passed to kernel.

[root@elm9m93 ~]# cat /proc/cmdline 
root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@16M

[root@elm9m93 ~]# /etc/init.d/kdump status
Kdump is not operational

[root@elm9m93 ~]# /etc/init.d/kdump start
Starting kdump:                                            [FAILED]

But reserving after 32M helps kdump to start its service.

[root@elm9m93 kdump]# cat /proc/cmdline 
root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@32M

[root@elm9m93 kdump]# /etc/init.d/kdump status
Kdump is operational

---------------------------------------------------------------------------
Operating System Info:

[root@elm9m93 ~]# uname -a
Linux elm9m93 2.6.31-rc5.2.el5rt #1 SMP PREEMPT RT Thu Aug 6 06:42:47 EDT 2009 x86_64 x86_64 x86_64
GNU/Linux

[root@elm9m93 ~]# cat /proc/meminfo 
MemTotal:       16470772 kB
MemFree:        15815640 kB
Buffers:           18924 kB
Cached:           153284 kB
SwapCached:            0 kB
Active:            41152 kB
Inactive:         155552 kB
Active(anon):      28552 kB
Inactive(anon):        0 kB
Active(file):      12600 kB
Inactive(file):   155552 kB
Unevictable:        4996 kB
Mlocked:            4996 kB
SwapTotal:       2096472 kB
SwapFree:        2096472 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         29560 kB
Mapped:            10680 kB
Slab:             323108 kB
SReclaimable:      20500 kB
SUnreclaim:       302608 kB
PageTables:         3284 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    10331856 kB
Committed_AS:      75108 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       79492 kB
VmallocChunk:   34359658459 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7920 kB
DirectMap2M:    16769024 kB
[root@elm9m93 ~]# 


[root@elm9m93 ~]# cat /proc/iomem 
00000000-0009cfff : System RAM
0009d000-0009ffff : reserved
000e0000-000fffff : reserved
00100000-cffbcdbf : System RAM
  01000000-013e29ed : Kernel code
  013e29ee-015e988f : Kernel data
  01689000-0174023f : Kernel bss
cffbcdc0-cffcffff : ACPI Tables
cffd0000-cfffffff : reserved
d0000000-d01fffff : PCI Bus 0000:02
  d0000000-d01fffff : 0000:02:00.0
d0200000-d02fffff : PCI Bus 0000:0c
  d0200000-d027ffff : 0000:0c:00.0
  d0280000-d02fffff : 0000:0c:00.1
d1000000-d6ffffff : PCI Bus 0000:0c
  d2000000-d3ffffff : 0000:0c:00.1
    d2000000-d3ffffff : bnx2x
  d4000000-d5ffffff : 0000:0c:00.0
    d4000000-d5ffffff : bnx2x
  d6000000-d67fffff : 0000:0c:00.1
    d6000000-d67fffff : bnx2x
  d6800000-d6ffffff : 0000:0c:00.0
    d6800000-d6ffffff : bnx2x
d7000000-d9ffffff : PCI Bus 0000:05
  d7000000-d9ffffff : PCI Bus 0000:06
    d8000000-d9ffffff : 0000:06:00.0
      d8000000-d9ffffff : bnx2
da000000-dcffffff : PCI Bus 0000:03
  da000000-dcffffff : PCI Bus 0000:04
    da000000-dbffffff : 0000:04:00.0
      da000000-dbffffff : bnx2
dd000000-deffffff : PCI Bus 0000:02
  defe0000-defeffff : 0000:02:00.0
    defe0000-defeffff : mpt
  deffc000-deffffff : 0000:02:00.0
    deffc000-deffffff : mpt
e0000000-efffffff : reserved
  e0000000-efffffff : pnp 00:0b
    e0000000-e0ffffff : PCI MMCONFIG 0 [00-0f]
f0000000-f7ffffff : PCI Bus 0000:01
  f0000000-f7ffffff : 0000:01:01.0
f8000000-f8ffffff : PCI Bus 0000:01
  f8000000-f800ffff : 0000:01:01.0
  f8020000-f803ffff : 0000:01:01.0
f9000000-f90003ff : 0000:00:1d.7
  f9000000-f90003ff : ehci_hcd
fe000000-fe01ffff : pnp 00:0b
  fe000000-fe01ffff : i5k_amb
fe020000-fe6fffff : pnp 00:0b
fe700000-fe7003ff : 0000:00:08.0
fe700400-febfffff : pnp 00:0b
fec00000-ffffffff : reserved
  fec00000-fec00fff : IOAPIC 0
  fec80000-fec80fff : IOAPIC 1
  fed1c000-fed1ffff : pnp 00:0b
  fee00000-fee00fff : Local APIC
  fff00000-ffffffff : pnp 00:0b
100000000-42fffffff : System RAM
[root@elm9m93 ~]# 

---------------------------------------------------------------------------
Hardwares tested:
LS21,HS21

---------------------------------------------------------------------------
Steps to reproduce:

o pass kernel param crashkernel=128M@16M
o start kdump service
   /etc/init.d/kdump start
=Comment: #1=================================================
SRIPATHI KODI <sripathik.com> - 
Alternatively, it is now enough to pass just "crashkernel=128M" as the parameter to the first
kernel. There is no need to pass the "@32M" option anymore. It is still required for older kernels,
though.

Looks like this is something RH needs to update in their kdump instructions
(http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html
??) Hence I'll request for mirroring this bug to RH.

Comment 1 IBM Bug Proxy 2009-08-28 02:20:26 UTC
------- Comment From gowrishankar.m.com 2009-08-27 22:15 EDT-------
From /proc/iomem report, it seems portion of memory is not
reserved (under 00100000-cffbcdbf : System RAM) for Crash
kernel.

Comment 2 IBM Bug Proxy 2009-10-19 12:50:41 UTC
------- Comment From sripathik.com 2009-10-19 08:48 EDT-------
We hit the same problem with debug kernel as well. Looks like recent kernels have gotten bigger and hence can't load the kdump kernel at low addresses. Debug kernel can't load crash kernel even at 32M. It needs to load at 48M. There is a message during bootup:

[root@elm9m98 ~]# dmesg |grep -i crashkernel
Command line: root=/dev/sda3 crashkernel=128M@32M console=tty1
console=ttyS1,19200
crashkernel reservation failed - memory is in use
Kernel command line: root=/dev/sda3 crashkernel=128M@32M console=tty1
console=ttyS1,19200

Since the kernel can figure out the offset by itself, one should not pass the offset while configuring kdump any more. I think RH should update http://rt.et.redhat.com/page/RHEL-RT_kdump/kexec and
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.1/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html
for MRG1.2.

------- Comment From sripathik.com 2009-10-19 08:49 EDT-------
*** Bug 55784 has been marked as a duplicate of this bug. ***

Comment 3 IBM Bug Proxy 2009-11-02 22:11:34 UTC
------- Comment From dvhltc.com 2009-11-02 17:06 EDT-------
Clark said he'll look into this and try to fix asap.

Comment 5 David Sommerseth 2009-11-25 16:47:49 UTC
No longer dependent on bug #536926, as this bug is related towards the 2.6.31 kernel, and not the 2.6.24.7 kernel MRG-1.2 will ship.

This bug is not relevant for the MRG-1.2 release in the current phase.  But will be important when we do ship a newer base kernel.

Comment 6 Clark Williams 2010-03-29 20:54:14 UTC
investigating correctness of the rt-setup-kdump script wrt .33 kernel

Comment 7 IBM Bug Proxy 2010-05-02 11:40:51 UTC
------- Comment From gowrishankar.m.com 2010-05-02 07:32 EDT-------
I have verified with the current mrg 1.3 alpha packages.

[root@elm9m96 ~]# uname -a
Linux elm9m96 2.6.33.2-rt13.12.el5rt #1 SMP PREEMPT RT Fri Apr 9 07:28:20 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@elm9m96 ~]# cat /proc/cmdline
root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@16M

[root@elm9m96 ~]# /etc/init.d/kdump start
Starting kdump:                                            [FAILED]

Crash memory is still not reserved if offset is set as 16M, where as with 32M offset
kdump works.

[root@elm9m96 ~]# /etc/init.d/kdump status
Kdump is operational

[root@elm9m96 ~]# cat /proc/cmdline
root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@32M

------- Comment From gowrishankar.m.com 2010-05-02 07:33 EDT-------
Used packages are:
[root@elm9m96 ~]# rpm -qa|grep rt-setup
rt-setup-1.6-3.el5rt

[root@elm9m96 ~]# rpm -qa|grep kdump
system-config-kdump-1.0.14-4.el5

Comment 8 IBM Bug Proxy 2010-05-03 15:20:40 UTC
------- Comment From dvhltc.com 2010-05-03 11:18 EDT-------
Clark is there a newer version of rt-setup we should be looking at?

Comment 9 IBM Bug Proxy 2010-05-26 18:11:46 UTC
------- Comment From gowrishankar.m.com 2010-05-26 14:01 EDT-------
(In reply to comment #16)
> Clark is there a newer version of rt-setup we should be looking at?

I am trying to understand the context of rt-setup-kdump. I played with
rt-setup-kdump (rt-setup-1.6-3.el5rt).

It just tries to update /etc/sysconfig/kdump and adds crashkernel entry
in /boot/grub/grub.conf (128M incase -v2 and 32M@128M in case of -v1).
So how is it going to address the offset issue of having 16M in our problem ?

Comment 10 IBM Bug Proxy 2010-05-26 18:31:00 UTC
------- Comment From dvhltc.com 2010-05-26 14:24 EDT-------
(In reply to comment #18)
> (In reply to comment #16)
> > Clark is there a newer version of rt-setup we should be looking at?
>
> I am trying to understand the context of rt-setup-kdump. I played with
>  rt-setup-kdump (rt-setup-1.6-3.el5rt).
>
> It just tries to update /etc/sysconfig/kdump and adds crashkernel entry
> in /boot/grub/grub.conf (128M incase -v2 and 32M@128M in case of -v1).
> So how is it going to address the offset issue of having 16M in our problem ?

My understanding is that the more recent kernels can autodetect where to reserve the memory for kdump. Explicitly setting it via the @16M or @32M can confuse things and cause failures. We expected that ensuring rt-setup specified only 128M and not 128M@XXM would resolve the problem.

Comment 11 IBM Bug Proxy 2010-05-31 10:40:42 UTC
------- Comment From gowrishankar.m.com 2010-05-31 06:36 EDT-------
I have verified rt-setup-kdump on rhel5.5 with both v1 and v2 kernels.
Crash kernel entries are added appropriately in grub.conf and kdump
works with them. I used rt-setup of version 1.6-3.el5rt .

Comment 12 IBM Bug Proxy 2010-05-31 11:41:00 UTC
------- Comment From amitarora.com 2010-05-31 07:35 EDT-------
Hello Redhat,

We have verified that the rt-setup-kdump works fine and we are able to get the dump with the newer kernels.

I hope there is a plan to make the required changes to the "Using kdump and kexec with the MRG Realtime kernel" section of the "Realtime Tuning Guide" for MRG 1.3 - with the release. If so, I will go ahead and close this bug. Please confirm.

For example, MRG1.2 has similar document at :
http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.2/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html

which states:

" Open the /etc/grub.conf file in your preferred text editor and add a crashkernel line to the boot kernel. This line takes the form:

crashkernel=[MB of RAM to reserve]M@[memory location]M

A typical crashkernel line would reserve 128 megabytes (128M) at 16 megabytes (16M), which is equivalant to the address 0x1000000:

crashkernel=128M@16M
"

Now, this document for MRG 1.3 should not suggest "@[mempry location]"  option.

Thanks!

Comment 13 IBM Bug Proxy 2011-06-08 21:10:54 UTC
------- Comment From gowrishankar.m.ibm.com 2010-05-02 07:32 EDT-------







------- Comment From gowrishankar.m.ibm.com 2010-05-02 07:33 EDT-------



------- Comment From gowrishankar.m.ibm.com 2010-05-26 14:01 EDT-------





------- Comment From gowrishankar.m.ibm.com 2010-05-31 06:36 EDT-------


Note You need to log in before you can comment on or make changes to this bug.