=Comment: #0================================================= GOWRISHANKAR MUTHUKRISHNAN <gowrishankar.m.com> - Problem Description: Kdump service fails to start in 2.6.31-rc5.2.el5rt when capture kernel memory is reserved after 16M i.e crashkernel=128M@16M is passed to kernel. [root@elm9m93 ~]# cat /proc/cmdline root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@16M [root@elm9m93 ~]# /etc/init.d/kdump status Kdump is not operational [root@elm9m93 ~]# /etc/init.d/kdump start Starting kdump: [FAILED] But reserving after 32M helps kdump to start its service. [root@elm9m93 kdump]# cat /proc/cmdline root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@32M [root@elm9m93 kdump]# /etc/init.d/kdump status Kdump is operational --------------------------------------------------------------------------- Operating System Info: [root@elm9m93 ~]# uname -a Linux elm9m93 2.6.31-rc5.2.el5rt #1 SMP PREEMPT RT Thu Aug 6 06:42:47 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux [root@elm9m93 ~]# cat /proc/meminfo MemTotal: 16470772 kB MemFree: 15815640 kB Buffers: 18924 kB Cached: 153284 kB SwapCached: 0 kB Active: 41152 kB Inactive: 155552 kB Active(anon): 28552 kB Inactive(anon): 0 kB Active(file): 12600 kB Inactive(file): 155552 kB Unevictable: 4996 kB Mlocked: 4996 kB SwapTotal: 2096472 kB SwapFree: 2096472 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 29560 kB Mapped: 10680 kB Slab: 323108 kB SReclaimable: 20500 kB SUnreclaim: 302608 kB PageTables: 3284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 10331856 kB Committed_AS: 75108 kB VmallocTotal: 34359738367 kB VmallocUsed: 79492 kB VmallocChunk: 34359658459 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 7920 kB DirectMap2M: 16769024 kB [root@elm9m93 ~]# [root@elm9m93 ~]# cat /proc/iomem 00000000-0009cfff : System RAM 0009d000-0009ffff : reserved 000e0000-000fffff : reserved 00100000-cffbcdbf : System RAM 01000000-013e29ed : Kernel code 013e29ee-015e988f : Kernel data 01689000-0174023f : Kernel bss cffbcdc0-cffcffff : ACPI Tables cffd0000-cfffffff : reserved d0000000-d01fffff : PCI Bus 0000:02 d0000000-d01fffff : 0000:02:00.0 d0200000-d02fffff : PCI Bus 0000:0c d0200000-d027ffff : 0000:0c:00.0 d0280000-d02fffff : 0000:0c:00.1 d1000000-d6ffffff : PCI Bus 0000:0c d2000000-d3ffffff : 0000:0c:00.1 d2000000-d3ffffff : bnx2x d4000000-d5ffffff : 0000:0c:00.0 d4000000-d5ffffff : bnx2x d6000000-d67fffff : 0000:0c:00.1 d6000000-d67fffff : bnx2x d6800000-d6ffffff : 0000:0c:00.0 d6800000-d6ffffff : bnx2x d7000000-d9ffffff : PCI Bus 0000:05 d7000000-d9ffffff : PCI Bus 0000:06 d8000000-d9ffffff : 0000:06:00.0 d8000000-d9ffffff : bnx2 da000000-dcffffff : PCI Bus 0000:03 da000000-dcffffff : PCI Bus 0000:04 da000000-dbffffff : 0000:04:00.0 da000000-dbffffff : bnx2 dd000000-deffffff : PCI Bus 0000:02 defe0000-defeffff : 0000:02:00.0 defe0000-defeffff : mpt deffc000-deffffff : 0000:02:00.0 deffc000-deffffff : mpt e0000000-efffffff : reserved e0000000-efffffff : pnp 00:0b e0000000-e0ffffff : PCI MMCONFIG 0 [00-0f] f0000000-f7ffffff : PCI Bus 0000:01 f0000000-f7ffffff : 0000:01:01.0 f8000000-f8ffffff : PCI Bus 0000:01 f8000000-f800ffff : 0000:01:01.0 f8020000-f803ffff : 0000:01:01.0 f9000000-f90003ff : 0000:00:1d.7 f9000000-f90003ff : ehci_hcd fe000000-fe01ffff : pnp 00:0b fe000000-fe01ffff : i5k_amb fe020000-fe6fffff : pnp 00:0b fe700000-fe7003ff : 0000:00:08.0 fe700400-febfffff : pnp 00:0b fec00000-ffffffff : reserved fec00000-fec00fff : IOAPIC 0 fec80000-fec80fff : IOAPIC 1 fed1c000-fed1ffff : pnp 00:0b fee00000-fee00fff : Local APIC fff00000-ffffffff : pnp 00:0b 100000000-42fffffff : System RAM [root@elm9m93 ~]# --------------------------------------------------------------------------- Hardwares tested: LS21,HS21 --------------------------------------------------------------------------- Steps to reproduce: o pass kernel param crashkernel=128M@16M o start kdump service /etc/init.d/kdump start =Comment: #1================================================= SRIPATHI KODI <sripathik.com> - Alternatively, it is now enough to pass just "crashkernel=128M" as the parameter to the first kernel. There is no need to pass the "@32M" option anymore. It is still required for older kernels, though. Looks like this is something RH needs to update in their kdump instructions (http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html ??) Hence I'll request for mirroring this bug to RH.
------- Comment From gowrishankar.m.com 2009-08-27 22:15 EDT------- From /proc/iomem report, it seems portion of memory is not reserved (under 00100000-cffbcdbf : System RAM) for Crash kernel.
------- Comment From sripathik.com 2009-10-19 08:48 EDT------- We hit the same problem with debug kernel as well. Looks like recent kernels have gotten bigger and hence can't load the kdump kernel at low addresses. Debug kernel can't load crash kernel even at 32M. It needs to load at 48M. There is a message during bootup: [root@elm9m98 ~]# dmesg |grep -i crashkernel Command line: root=/dev/sda3 crashkernel=128M@32M console=tty1 console=ttyS1,19200 crashkernel reservation failed - memory is in use Kernel command line: root=/dev/sda3 crashkernel=128M@32M console=tty1 console=ttyS1,19200 Since the kernel can figure out the offset by itself, one should not pass the offset while configuring kdump any more. I think RH should update http://rt.et.redhat.com/page/RHEL-RT_kdump/kexec and http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.1/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html for MRG1.2. ------- Comment From sripathik.com 2009-10-19 08:49 EDT------- *** Bug 55784 has been marked as a duplicate of this bug. ***
------- Comment From dvhltc.com 2009-11-02 17:06 EDT------- Clark said he'll look into this and try to fix asap.
No longer dependent on bug #536926, as this bug is related towards the 2.6.31 kernel, and not the 2.6.24.7 kernel MRG-1.2 will ship. This bug is not relevant for the MRG-1.2 release in the current phase. But will be important when we do ship a newer base kernel.
investigating correctness of the rt-setup-kdump script wrt .33 kernel
------- Comment From gowrishankar.m.com 2010-05-02 07:32 EDT------- I have verified with the current mrg 1.3 alpha packages. [root@elm9m96 ~]# uname -a Linux elm9m96 2.6.33.2-rt13.12.el5rt #1 SMP PREEMPT RT Fri Apr 9 07:28:20 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [root@elm9m96 ~]# cat /proc/cmdline root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@16M [root@elm9m96 ~]# /etc/init.d/kdump start Starting kdump: [FAILED] Crash memory is still not reserved if offset is set as 16M, where as with 32M offset kdump works. [root@elm9m96 ~]# /etc/init.d/kdump status Kdump is operational [root@elm9m96 ~]# cat /proc/cmdline root=/dev/sda3 console=ttyS1,19200 crashkernel=128M@32M ------- Comment From gowrishankar.m.com 2010-05-02 07:33 EDT------- Used packages are: [root@elm9m96 ~]# rpm -qa|grep rt-setup rt-setup-1.6-3.el5rt [root@elm9m96 ~]# rpm -qa|grep kdump system-config-kdump-1.0.14-4.el5
------- Comment From dvhltc.com 2010-05-03 11:18 EDT------- Clark is there a newer version of rt-setup we should be looking at?
------- Comment From gowrishankar.m.com 2010-05-26 14:01 EDT------- (In reply to comment #16) > Clark is there a newer version of rt-setup we should be looking at? I am trying to understand the context of rt-setup-kdump. I played with rt-setup-kdump (rt-setup-1.6-3.el5rt). It just tries to update /etc/sysconfig/kdump and adds crashkernel entry in /boot/grub/grub.conf (128M incase -v2 and 32M@128M in case of -v1). So how is it going to address the offset issue of having 16M in our problem ?
------- Comment From dvhltc.com 2010-05-26 14:24 EDT------- (In reply to comment #18) > (In reply to comment #16) > > Clark is there a newer version of rt-setup we should be looking at? > > I am trying to understand the context of rt-setup-kdump. I played with > rt-setup-kdump (rt-setup-1.6-3.el5rt). > > It just tries to update /etc/sysconfig/kdump and adds crashkernel entry > in /boot/grub/grub.conf (128M incase -v2 and 32M@128M in case of -v1). > So how is it going to address the offset issue of having 16M in our problem ? My understanding is that the more recent kernels can autodetect where to reserve the memory for kdump. Explicitly setting it via the @16M or @32M can confuse things and cause failures. We expected that ensuring rt-setup specified only 128M and not 128M@XXM would resolve the problem.
------- Comment From gowrishankar.m.com 2010-05-31 06:36 EDT------- I have verified rt-setup-kdump on rhel5.5 with both v1 and v2 kernels. Crash kernel entries are added appropriately in grub.conf and kdump works with them. I used rt-setup of version 1.6-3.el5rt .
------- Comment From amitarora.com 2010-05-31 07:35 EDT------- Hello Redhat, We have verified that the rt-setup-kdump works fine and we are able to get the dump with the newer kernels. I hope there is a plan to make the required changes to the "Using kdump and kexec with the MRG Realtime kernel" section of the "Realtime Tuning Guide" for MRG 1.3 - with the release. If so, I will go ahead and close this bug. Please confirm. For example, MRG1.2 has similar document at : http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.2/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Using_kdump_and_kexec_with_the_RT_kernel.html which states: " Open the /etc/grub.conf file in your preferred text editor and add a crashkernel line to the boot kernel. This line takes the form: crashkernel=[MB of RAM to reserve]M@[memory location]M A typical crashkernel line would reserve 128 megabytes (128M) at 16 megabytes (16M), which is equivalant to the address 0x1000000: crashkernel=128M@16M " Now, this document for MRG 1.3 should not suggest "@[mempry location]" option. Thanks!
------- Comment From gowrishankar.m.ibm.com 2010-05-02 07:32 EDT------- ------- Comment From gowrishankar.m.ibm.com 2010-05-02 07:33 EDT------- ------- Comment From gowrishankar.m.ibm.com 2010-05-26 14:01 EDT------- ------- Comment From gowrishankar.m.ibm.com 2010-05-31 06:36 EDT-------