|Summary:||To kexec the pxeboot image can cause system hang.|
|Product:||Red Hat Enterprise Linux 6||Reporter:||Kirby Zhou <kirbyzhou>|
|Component:||kexec-tools||Assignee:||Neil Horman <nhorman>|
|Status:||CLOSED NOTABUG||QA Contact:||Red Hat Kernel QE team <kernel-qe>|
|Fixed In Version:||Doc Type:||Bug Fix|
|Doc Text:||Story Points:||---|
|Last Closed:||2010-07-14 12:14:29 UTC||Type:||---|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
Description Kirby Zhou 2010-05-04 04:44:21 UTC
Description of problem: With a RHEL6-beta box, I copy the vmlinuz and initrd.img from dvd/images/pxeboot and run kexec with them, then system hang. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: kexec --initrd=/boot/install/initrd.img /boot/install/vmlinuz Actual results: "Starting the new kernel", then hang Expected results: boot into the installation interface. Additional info: IBM X3650-M2 with E5520*2, 48G RAM.
Comment 2 RHEL Product and Program Management 2010-05-04 06:17:12 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red Hat Enterprise Linux major release. Product Management has requested further review of this request by Red Hat Engineering, for potential inclusion in a Red Hat Enterprise Linux Major release. This request is not yet committed for inclusion.
Comment 3 Neil Horman 2010-05-04 10:40:03 UTC
This may be a duplicate of another bug I have how are you crashing the kernel after you load the panic kernel? Also, why are you manually running kexec rather than using the service script? Can you provide a full console log of the reproducing environment during a crash? Thanks!
Comment 4 Kirby Zhou 2010-05-04 11:33:28 UTC
I donot what do you mean about "using the service script". I just want to use kexec to reinstall a RHEL6 system through ssh connection, but it failed. To find the reason, I tested the simplest situation: A brandly new installed RHEL6 box (x86_64), just do kexec with simplest arguments, watching it from the console. The system is successfully shutdown by kexec, but hang after the message "Starting the new kernel"
Comment 5 Neil Horman 2010-05-04 21:50:19 UTC
Ok, so You're not using kexec as part of the kdump service. In that case can you try loading the kdump kernel first (via kexec -l), then running the kexec -e to execute it? Also, if you're just trying to get a RHEL6 system installed over ssh, there are far simpler ways to do it. Why not just proxy a vnc session over an ssh tunnel?
Comment 6 Kirby Zhou 2010-05-05 03:24:48 UTC
" Also, if you're just trying to get a RHEL6 system installed over ssh, there are far simpler ways to do it. Why not just proxy a vnc session over an ssh tunnel? " I think you misunderstanding my requirements. I want to automaticly reinstall a system WITHOUT touching the keyboard, the monitor, and cdrom drive of the PC, and donot touch any part of the DHCP system. Starting a vnc session is easy, the diffculty is how to boot the installer.
Comment 7 Neil Horman 2010-05-05 15:17:38 UTC
Ah, I see, you're trying to pxeboot a system from a running kernel to re-install the system. I get it now. My previous suggestion still stands though, please try using the kexec -l command to load the kernel, then follow it with kexec -e to actually execute the new kernel.
Comment 8 Kirby Zhou 2010-05-06 02:35:55 UTC
'kexec -l' which kernel? the pxeboot kernel or something else?
Comment 9 Kirby Zhou 2010-05-06 03:14:58 UTC
kexec -l vmlinuz --initrd=initrd.img kexec -e Just a "Starting the new kernel", then hang. This time, system is not automatically shut down before "Starting the new kernel"
Comment 10 Neil Horman 2010-05-06 15:15:50 UTC
ok, can you try attaching a serial console or a monitor and enabling earlyprintk in the kernel (http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/kernel-parameters.txt;h=839b21b0699ac10a1991c47455cddd05ced6491b;hb=HEAD) That will give us a better idea of how far its getting during the boot process Also, if you're trying this from an X session, make sure that you're just on a VT so that we can be sure the video mode is such that you can get console output on your monitor.
Comment 11 Kirby Zhou 2010-05-07 17:49:08 UTC
I donot use X. [root@djt-18-97 ~]# cat /proc/cmdline ro root=/dev/mapper/vgroot-lvroot rd_LVM_LV=vgroot/lvroot rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga [root@djt-18-97 ~]# kexec --initrd=/boot/install/initrd.img -l /boot/install/vmlinuz --command-line='earlyprintk=vga' [root@djt-18-97 ~]# kexec -e Wait for 1 minute with nothing output to screen, then "Starting the new kernel" and HANG.
Comment 12 Neil Horman 2010-05-07 18:15:48 UTC
That means you're not getting into the new kernel at all. The system is hanging sometime prior to jumping into the new kernel. I wonder if the problem here isn't that we're loading the kdump kernel into a location greater then 4GB. Given that you have 48GB of ram, I can imagine that is entirely possible. And since you're using the facility directly, instead of for a crash kernel (as its nominally used), I would not be at all suprised if thats happening. Unfortunately, its difficult to tell with -l where the kernel / initramfs will reside becuase the memory is allocated dynamically. And if thats the problem I'm not sure theres anythign we'll be able to do about it. The test for this should be pretty straightforward. You should try either of the following: 1) boot the system with mem=4G, and attempt your test above OR 1) boot with this on the command line: crashkernel=128M@1G 2)load the pxeboot image using the -p option rather than the -l option 3) crash the kernel with this command: echo c > /proc/sysrq-trigger I would prefer the second method as it guarantees the placement of the kenrel and initramfs in memory. But either should work. If you run these tests and you manage to boot your pxe kernel, that should be indicative of the fact that straightforward use of the kexec utility in this system is placing the kernel in a bad location.
Comment 13 Kirby Zhou 2010-05-10 02:17:50 UTC
Failed with method 2 128M@1G failed, 256M@16 also failed with the same message. ]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz --command-line='earlyprintk=vga' Could not find a free area of memory of 395000 bytes... ]# dmesg | fgrep crash Reserving 256MB of memory at 1024MB for crashkernel (System RAM: 51200MB) Kernel command line: ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=256M@1G crash memory driver: version 1.0 ]# free total used free shared buffers cached Mem: 49290352 1451012 47839340 0 58816 278200 -/+ buffers/cache: 1113996 48176356 Swap: 8388600 0 8388600
Comment 14 Neil Horman 2010-05-10 10:43:55 UTC
Sorry, I forogot this system profile, you'll actually need to reserve a larger amount of ram (more along the lines of 512M-1024M to get all the space you need.
Comment 15 Kirby Zhou 2010-05-12 03:44:42 UTC
]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz --command-line='earlyprintk=vga' Could not find a free area of memory of 395000 bytes... ]# cat /proc/cmdline ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=768M@1G mem=3500M
Comment 16 Cong Wang 2010-05-13 04:06:13 UTC
(In reply to comment #15) > ]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz > --command-line='earlyprintk=vga' > Could not find a free area of memory of 395000 bytes... > ]# cat /proc/cmdline > ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 > KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=768M@1G > mem=3500M How does your /proc/iomem look like? Perhaps the memory your reserved for crashkernel failed this time.
Comment 17 Kirby Zhou 2010-05-13 04:37:52 UTC
Withou crashkernel= and mem= I think there is no problem with 768M@1G 00100000-7d90bfff : System RAM 01000000-014c9f4e : Kernel code 014c9f4f-01899b2f : Kernel data 019c2000-01c1bd17 : Kernel bss ]# cat /proc/iomem 00000000-0009c3ff : System RAM 0009c400-0009ffff : reserved 000e0000-000fffff : reserved 00100000-7d90bfff : System RAM 01000000-014c9f4e : Kernel code 014c9f4f-01899b2f : Kernel data 019c2000-01c1bd17 : Kernel bss 7d90c000-7d9bbfff : reserved 7d9bc000-7f68efff : System RAM 7f68f000-7f6defff : reserved 7f6df000-7f7defff : ACPI Non-volatile Storage 7f7df000-7f7fefff : ACPI Tables 7f7ff000-7f7fffff : System RAM 7f800000-8fffffff : reserved 80000000-8fffffff : PCI MMCONFIG 0 [00-ff] 80000000-8fffffff : pnp 00:0a 90000000-900fffff : PCI Bus 0000:1a 90000000-9003ffff : 0000:1a:00.0 90100000-902fffff : PCI Bus 0000:01 90300000-904fffff : PCI Bus 0000:01 92000000-95ffffff : PCI Bus 0000:0b 92000000-93ffffff : 0000:0b:00.0 92000000-93ffffff : bnx2 94000000-95ffffff : 0000:0b:00.1 94000000-95ffffff : bnx2 96000000-96ffffff : PCI Bus 0000:06 96000000-96ffffff : PCI Bus 0000:07 96000000-96ffffff : 0000:07:00.0 96000000-96ffffff : matroxfb FB 97000000-978fffff : PCI Bus 0000:06 97000000-978fffff : PCI Bus 0000:07 97000000-977fffff : 0000:07:00.0 97800000-97803fff : 0000:07:00.0 97800000-97803fff : matroxfb MMIO 97900000-979fffff : PCI Bus 0000:1a 97900000-9793ffff : 0000:1a:00.0 97900000-9793ffff : megasas: LSI 97940000-97943fff : 0000:1a:00.0 97940000-97943fff : megasas: LSI 97a00000-97a03fff : 0000:00:16.0 97a00000-97a03fff : ioatdma 97a04000-97a07fff : 0000:00:16.1 97a04000-97a07fff : ioatdma 97a08000-97a0bfff : 0000:00:16.2 97a08000-97a0bfff : ioatdma 97a0c000-97a0ffff : 0000:00:16.3 97a0c000-97a0ffff : ioatdma 97a10000-97a13fff : 0000:00:16.4 97a10000-97a13fff : ioatdma 97a14000-97a17fff : 0000:00:16.5 97a14000-97a17fff : ioatdma 97a18000-97a1bfff : 0000:00:16.6 97a18000-97a1bfff : ioatdma 97a1c000-97a1ffff : 0000:00:16.7 97a1c000-97a1ffff : ioatdma 97a21000-97a213ff : 0000:00:1d.7 97a21000-97a213ff : ehci_hcd 97a21400-97a217ff : 0000:00:1a.7 97a21400-97a217ff : ehci_hcd 97a21800-97a218ff : 0000:00:1f.3 fc000000-fcffffff : pnp 00:0a fe710000-fe711fff : pnp 00:0a fe800000-fe9fffff : pnp 00:0a fea00000-feafffff : pnp 00:0a feb00000-febfffff : pnp 00:0a fec00000-fec00fff : IOAPIC 0 fec80000-fec80fff : IOAPIC 1 fed00000-fed003ff : HPET 0 fed00000-fed003ff : pnp 00:05 fed1c000-fed1ffff : reserved fed1c000-fed1ffff : pnp 00:0a fee00000-feefffff : pnp 00:0a fee00000-fee00fff : Local APIC ff800000-ffffffff : reserved ffc00000-ffffffff : pnp 00:0a 100000000-c7fffffff : System RAM
Comment 18 Cong Wang 2010-05-13 05:04:42 UTC
(In reply to comment #17) > Withou crashkernel= and mem= > I think there is no problem with 768M@1G > > ]# cat /proc/iomem > > 00000000-0009c3ff : System RAM > 0009c400-0009ffff : reserved > 000e0000-000fffff : reserved > 00100000-7d90bfff : System RAM > 01000000-014c9f4e : Kernel code > 014c9f4f-01899b2f : Kernel data > 019c2000-01c1bd17 : Kernel bss > 7d90c000-7d9bbfff : reserved > 7d9bc000-7f68efff : System RAM 28M here. > 7f7ff000-7f7fffff : System RAM Only 4M here. > 100000000-c7fffffff : System RAM 42G@4G here. So probably kexec loaded the kernel into the last memory area which is @4G. I think this is similar to Bug 580843, currently there are some problems if we load kernel into memory higher than 4G. Vitaly is working on this.
Comment 19 Cong Wang 2010-05-13 06:04:04 UTC
Note, comment #18 is for your 'kexec -l' case. For 'kexec -p' case, that should be a bug of kexec, I can reproduce it here, 'service kdump start' failed too. I am checking the code.
Comment 20 Cong Wang 2010-05-26 09:59:38 UTC
Hi, does appending "nomodeset" to your kernel commandline help? I noticed KMS doesn't work with kdump.
Comment 21 Kirby Zhou 2010-05-27 15:47:29 UTC
Created attachment 417292 [details] nomodeset do not help. nomodeset do not help. ~]# cat /proc/cmdline ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga nomodeset ]# kexec --initrd=/boot/install/initrd.img /boot/install/vmlinuz --command-line='earlyprintk=vga nomodeset'
Comment 22 Neil Horman 2010-07-06 18:10:15 UTC
can you try reserving the memory without specifying a location? That should give the kernel the freedom to find a sufficiently sized hole
Comment 23 Kirby Zhou 2010-07-14 11:51:37 UTC
It seems workable now. #] kexec --initrd=/boot/initramfs-2.6.32-37.el6.x86_64.img --command-line='ro root=/dev/mapper/vgroot-lvroot rd_LVM_LV=vgroot/lvroot rd_LVM_LV=vgroot/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet' /boot/vmlinuz-2.6.32-37.el6.x86_64 After a short while, the system is up again.
Comment 24 Neil Horman 2010-07-14 12:14:29 UTC
Ok, I expect you are somehow trampling on system space that the kernel or hardware is using for something else (usually ACPI space). Since we don't really use the precise locaiton syntax anymore for just this reason, I think we can close this as working