Bugzilla will be upgraded to version 5.0 on a still to be determined date in the near future. The original upgrade date has been delayed.
Bug 588609 - To kexec the pxeboot image can cause system hang.
To kexec the pxeboot image can cause system hang.
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kexec-tools (Show other bugs)
6.0
All Linux
low Severity medium
: rc
: ---
Assigned To: Neil Horman
Red Hat Kernel QE team
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2010-05-04 00:44 EDT by Kirby Zhou
Modified: 2010-07-14 08:14 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-07-14 08:14:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
nomodeset do not help. (35.68 KB, image/gif)
2010-05-27 11:47 EDT, Kirby Zhou
no flags Details

  None (edit)
Description Kirby Zhou 2010-05-04 00:44:21 EDT
Description of problem:

With a RHEL6-beta box, I copy the vmlinuz and initrd.img from dvd/images/pxeboot and run kexec with them, then system hang.

Version-Release number of selected component (if applicable):


How reproducible:

100%

Steps to Reproduce:

kexec --initrd=/boot/install/initrd.img /boot/install/vmlinuz
  
Actual results:

 "Starting the new kernel", then hang

Expected results:

  boot into the installation interface.

Additional info:

IBM X3650-M2 with E5520*2, 48G RAM.
Comment 2 RHEL Product and Program Management 2010-05-04 02:17:12 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 3 Neil Horman 2010-05-04 06:40:03 EDT
This may be a duplicate of another bug I have
how are you crashing the kernel after you load the panic kernel?  Also, why are you manually running kexec rather than using the service script?  Can you provide a full console log of the reproducing environment during a crash?  Thanks!
Comment 4 Kirby Zhou 2010-05-04 07:33:28 EDT
I donot what do you mean about "using the service script".
I just want to use kexec to reinstall a RHEL6 system through ssh connection, but it failed. 
To find the reason, I tested the simplest situation: A brandly new installed RHEL6 box (x86_64), just do kexec with simplest arguments, watching it from the console. The system is successfully shutdown by kexec, but hang after the message "Starting the new kernel"
Comment 5 Neil Horman 2010-05-04 17:50:19 EDT
Ok, so You're not using kexec as part of the kdump service.  In that case can you try loading the kdump kernel first (via kexec -l), then running the kexec -e to execute it?  Also, if you're just trying to get a RHEL6 system installed over ssh, there are far simpler ways to do it.  Why not just proxy a vnc session over an ssh tunnel?
Comment 6 Kirby Zhou 2010-05-04 23:24:48 EDT
" Also, if you're just trying to get a RHEL6 system installed
over ssh, there are far simpler ways to do it.  Why not just proxy a vnc
session over an ssh tunnel? "

I think you misunderstanding my requirements. I want to automaticly reinstall a system WITHOUT touching the keyboard, the monitor, and cdrom drive of the PC, and donot touch any part of the DHCP system. 
Starting a vnc session is easy, the diffculty is how to boot the installer.
Comment 7 Neil Horman 2010-05-05 11:17:38 EDT
Ah, I see, you're trying to pxeboot a system from a running kernel to re-install the system.  I get it now.  My previous suggestion still stands though, please try using the kexec -l command to load the kernel, then follow it with kexec -e to actually execute the new kernel.
Comment 8 Kirby Zhou 2010-05-05 22:35:55 EDT
'kexec -l' which kernel? the pxeboot kernel or something else?
Comment 9 Kirby Zhou 2010-05-05 23:14:58 EDT
kexec -l vmlinuz --initrd=initrd.img
kexec -e

Just a "Starting the new kernel", then hang.

This time, system is not automatically shut down before "Starting the new kernel"
Comment 10 Neil Horman 2010-05-06 11:15:50 EDT
ok, can you try attaching a serial console or a monitor  and enabling earlyprintk in the kernel (http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/kernel-parameters.txt;h=839b21b0699ac10a1991c47455cddd05ced6491b;hb=HEAD)

That will give us a better idea of how far its getting during the boot process

Also, if you're trying this from an X session, make sure that you're just on a VT so that we can be sure the video mode is such that you can get console output on your monitor.
Comment 11 Kirby Zhou 2010-05-07 13:49:08 EDT
I donot use X.

[root@djt-18-97 ~]# cat /proc/cmdline 
ro root=/dev/mapper/vgroot-lvroot rd_LVM_LV=vgroot/lvroot rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga

[root@djt-18-97 ~]# kexec --initrd=/boot/install/initrd.img -l /boot/install/vmlinuz  --command-line='earlyprintk=vga'

[root@djt-18-97 ~]# kexec -e

Wait for 1 minute with nothing output to screen, then  "Starting the new kernel" and HANG.
Comment 12 Neil Horman 2010-05-07 14:15:48 EDT
That means you're not getting into the new kernel at all.  The system is hanging sometime prior to jumping into the new kernel.

I wonder if the problem here isn't that we're loading the kdump kernel into a location greater then 4GB.  Given that you have 48GB of ram, I can imagine that is entirely possible.  And since you're using the facility directly, instead of for a crash kernel (as its nominally used), I would not be at all suprised if thats happening.  Unfortunately, its difficult to tell with -l where the kernel / initramfs will reside becuase the memory is allocated dynamically.  And if thats the problem I'm not sure theres anythign we'll be able to do about it.

The test for this should be pretty straightforward.  You should try either of the following:

1) boot the system with mem=4G, and attempt your test above

OR

1) boot with this on the command line:
crashkernel=128M@1G
2)load the pxeboot image using the -p option rather than the -l option
3) crash the kernel with this command:
echo c > /proc/sysrq-trigger

I would prefer the second method as it guarantees the placement of the kenrel and initramfs in memory.  But either should work.

If you run these tests and you manage to boot your pxe kernel, that should be indicative of the fact that straightforward use of the kexec utility in this system is placing the kernel in a bad location.
Comment 13 Kirby Zhou 2010-05-09 22:17:50 EDT
Failed with method 2

128M@1G failed, 256M@16 also failed with the same message.

]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz  --command-line='earlyprintk=vga'
Could not find a free area of memory of 395000 bytes...

]# dmesg | fgrep crash
Reserving 256MB of memory at 1024MB for crashkernel (System RAM: 51200MB)
Kernel command line: ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=256M@1G
crash memory driver: version 1.0

]# free
             total       used       free     shared    buffers     cached
Mem:      49290352    1451012   47839340          0      58816     278200
-/+ buffers/cache:    1113996   48176356
Swap:      8388600          0    8388600
Comment 14 Neil Horman 2010-05-10 06:43:55 EDT
Sorry, I forogot this system profile, you'll actually need to reserve a larger amount of ram (more along the lines of 512M-1024M to get all the space you need.
Comment 15 Kirby Zhou 2010-05-11 23:44:42 EDT
]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz  --command-line='earlyprintk=vga'
Could not find a free area of memory of 395000 bytes...
]# cat /proc/cmdline 
ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=768M@1G mem=3500M
Comment 16 Cong Wang 2010-05-13 00:06:13 EDT
(In reply to comment #15)
> ]# kexec --initrd=/boot/install/initrd.img -p /boot/install/vmlinuz 
> --command-line='earlyprintk=vga'
> Could not find a free area of memory of 395000 bytes...
> ]# cat /proc/cmdline 
> ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16
> KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga crashkernel=768M@1G
> mem=3500M    

How does your /proc/iomem look like?
Perhaps the memory your reserved for crashkernel failed this time.
Comment 17 Kirby Zhou 2010-05-13 00:37:52 EDT
Withou crashkernel= and mem=
I think there is no problem with 768M@1G

00100000-7d90bfff : System RAM
  01000000-014c9f4e : Kernel code
  014c9f4f-01899b2f : Kernel data
  019c2000-01c1bd17 : Kernel bss


]# cat /proc/iomem 

00000000-0009c3ff : System RAM
0009c400-0009ffff : reserved
000e0000-000fffff : reserved
00100000-7d90bfff : System RAM
  01000000-014c9f4e : Kernel code
  014c9f4f-01899b2f : Kernel data
  019c2000-01c1bd17 : Kernel bss
7d90c000-7d9bbfff : reserved
7d9bc000-7f68efff : System RAM
7f68f000-7f6defff : reserved
7f6df000-7f7defff : ACPI Non-volatile Storage
7f7df000-7f7fefff : ACPI Tables
7f7ff000-7f7fffff : System RAM
7f800000-8fffffff : reserved
  80000000-8fffffff : PCI MMCONFIG 0 [00-ff]
    80000000-8fffffff : pnp 00:0a
90000000-900fffff : PCI Bus 0000:1a
  90000000-9003ffff : 0000:1a:00.0
90100000-902fffff : PCI Bus 0000:01
90300000-904fffff : PCI Bus 0000:01
92000000-95ffffff : PCI Bus 0000:0b
  92000000-93ffffff : 0000:0b:00.0
    92000000-93ffffff : bnx2
  94000000-95ffffff : 0000:0b:00.1
    94000000-95ffffff : bnx2
96000000-96ffffff : PCI Bus 0000:06
  96000000-96ffffff : PCI Bus 0000:07
    96000000-96ffffff : 0000:07:00.0
      96000000-96ffffff : matroxfb FB
97000000-978fffff : PCI Bus 0000:06
  97000000-978fffff : PCI Bus 0000:07
    97000000-977fffff : 0000:07:00.0
    97800000-97803fff : 0000:07:00.0
      97800000-97803fff : matroxfb MMIO
97900000-979fffff : PCI Bus 0000:1a
  97900000-9793ffff : 0000:1a:00.0
    97900000-9793ffff : megasas: LSI
  97940000-97943fff : 0000:1a:00.0
    97940000-97943fff : megasas: LSI
97a00000-97a03fff : 0000:00:16.0
  97a00000-97a03fff : ioatdma
97a04000-97a07fff : 0000:00:16.1
  97a04000-97a07fff : ioatdma
97a08000-97a0bfff : 0000:00:16.2
  97a08000-97a0bfff : ioatdma
97a0c000-97a0ffff : 0000:00:16.3
  97a0c000-97a0ffff : ioatdma
97a10000-97a13fff : 0000:00:16.4
  97a10000-97a13fff : ioatdma
97a14000-97a17fff : 0000:00:16.5
  97a14000-97a17fff : ioatdma
97a18000-97a1bfff : 0000:00:16.6
  97a18000-97a1bfff : ioatdma
97a1c000-97a1ffff : 0000:00:16.7
  97a1c000-97a1ffff : ioatdma
97a21000-97a213ff : 0000:00:1d.7
  97a21000-97a213ff : ehci_hcd
97a21400-97a217ff : 0000:00:1a.7
  97a21400-97a217ff : ehci_hcd
97a21800-97a218ff : 0000:00:1f.3
fc000000-fcffffff : pnp 00:0a
fe710000-fe711fff : pnp 00:0a
fe800000-fe9fffff : pnp 00:0a
fea00000-feafffff : pnp 00:0a
feb00000-febfffff : pnp 00:0a
fec00000-fec00fff : IOAPIC 0
fec80000-fec80fff : IOAPIC 1
fed00000-fed003ff : HPET 0
  fed00000-fed003ff : pnp 00:05
fed1c000-fed1ffff : reserved
  fed1c000-fed1ffff : pnp 00:0a
fee00000-feefffff : pnp 00:0a
  fee00000-fee00fff : Local APIC
ff800000-ffffffff : reserved
  ffc00000-ffffffff : pnp 00:0a
100000000-c7fffffff : System RAM
Comment 18 Cong Wang 2010-05-13 01:04:42 EDT
(In reply to comment #17)
> Withou crashkernel= and mem=
> I think there is no problem with 768M@1G
>  
> ]# cat /proc/iomem 
> 
> 00000000-0009c3ff : System RAM
> 0009c400-0009ffff : reserved
> 000e0000-000fffff : reserved
> 00100000-7d90bfff : System RAM
>   01000000-014c9f4e : Kernel code
>   014c9f4f-01899b2f : Kernel data
>   019c2000-01c1bd17 : Kernel bss
> 7d90c000-7d9bbfff : reserved
> 7d9bc000-7f68efff : System RAM

28M here.

> 7f7ff000-7f7fffff : System RAM

Only 4M here.

> 100000000-c7fffffff : System RAM    

42G@4G here.

So probably kexec loaded the kernel into the last memory area which is @4G. I think this is similar to Bug 580843, currently there are some problems if we load kernel into memory higher than 4G. Vitaly is working on this.
Comment 19 Cong Wang 2010-05-13 02:04:04 EDT
Note, comment #18 is for your 'kexec -l' case. For 'kexec -p' case, that should be a bug of kexec, I can reproduce it here, 'service kdump start' failed too. I am checking the code.
Comment 20 Cong Wang 2010-05-26 05:59:38 EDT
Hi,

does appending "nomodeset" to your kernel commandline help?
I noticed KMS doesn't work with kdump.
Comment 21 Kirby Zhou 2010-05-27 11:47:29 EDT
Created attachment 417292 [details]
nomodeset do not help.

nomodeset do not help.


~]# cat /proc/cmdline 
ro root=/dev/mapper/vgroot-lvroot LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet earlyprintk=vga nomodeset

]# kexec --initrd=/boot/install/initrd.img /boot/install/vmlinuz  --command-line='earlyprintk=vga nomodeset'
Comment 22 Neil Horman 2010-07-06 14:10:15 EDT
can you try reserving the memory without specifying a location?  That should give the kernel the freedom to find a sufficiently sized hole
Comment 23 Kirby Zhou 2010-07-14 07:51:37 EDT
It seems workable now.


#] kexec --initrd=/boot/initramfs-2.6.32-37.el6.x86_64.img --command-line='ro root=/dev/mapper/vgroot-lvroot rd_LVM_LV=vgroot/lvroot rd_LVM_LV=vgroot/lvswap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet' /boot/vmlinuz-2.6.32-37.el6.x86_64 

After a short while, the system is up again.
Comment 24 Neil Horman 2010-07-14 08:14:29 EDT
Ok, I expect you are somehow trampling on system space that the kernel or hardware is using for something else (usually ACPI space).  Since we don't really use the precise locaiton syntax anymore for just this reason, I think we can close this as working

Note You need to log in before you can comment on or make changes to this bug.