what is the kernel command line, please show /boot/grub/grub.conf

# cat /boot/grub/grub.conf
default=1
timeout=0
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-220.2.1.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=LABEL=_/ crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us
        initrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img
title koji-image-RHEL-6.2-build-3896128-201112151048 (2.6.32-220.el6.x86_64)
        root (hd0)
        kernel /boot/vmlinuz-2.6.32-220.el6.x86_64 ro root=LABEL=_/
        initrd /boot/initramfs-2.6.32-220.el6.x86_64.img

Umm, re-reading my Steps to reproduce, it needs the

  s/it will fail to upgrade afterwards/it will fail to boot afterwards

change.

can you attach the output of:

# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img

and

# lsinitrd /boot/initramfs-2.6.32-220.el6.x86_64.img

Created attachment 550899 [details]
lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img

Created attachment 550901 [details]
lsinitrd /boot/initramfs-2.6.32-220.el6.x86_64.img

Fresh install... kernel-2.6.32-220.el6.x86_64 was running

I did a

# yum install kernel

so, I see that depmod was not run (or failed??) before dracut was called in the kernel %posttrans

/lib/modules/2.6.32-220.2.1.el6.x86_64/modules.dep does not exists

this points to 

/sbin/new-kernel-pkg

after doing

# depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64 2.6.32-220.2.1.el6.x86_64

# dracut --force '' 
2.6.32-220.2.1.el6.x86_64

all was fine after a reboot

I see the same problems for F16 sometime. Missing depmod, missing initramfs line, etc..

I've tried to upgrade the kernel a couple more times, with the following results:

- SELinux does not play any role;
- rpm -i and yum localinstall place the modules to the ramdisk just fine;
- yum upgrade and yum install do not place the modules to the ramdisk.

I was testing with

# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l

after the upgrade / installation, then did

# rpm -e kernel-2.6.32-220.2.1.el6.x86_64

and started anew with the next type of upgrade. Things are deterministic.

I've run yum localinstall kernel-2.6.32-220.2.1.el6.x86_64.rpm and yum upgrade kernel under strace -f and started diffing the log, after (manually) modifying the pids in the log of respective processes to match.

The difference is at

 1206  execve("/sbin/depmod", ["depmod", "-ae", "-F", "/boot/System.map-2.6.32-220.2.1."..., "2.6.32-220.2.1.el6.x86_64"], [/* 27 vars */]) 
= 0
@@ -160548,7 +237103,7 @@
 1206  read(7, "\213O\330H\201\301\260\0\0\0\351\r\377\377\377\220D\213K\330A\201\301\260\0\0\0\351\21\377\377\377"..., 16384) = 16384
 1206  read(7, "\0\0\0\0L\213f\20I\211\365L\215u\200L\215}\260H\211\373\350\0\0\0\0I\215u0L"..., 32768) = 32768

[...]

 1206  read(7, "\351'\377\377\377A\213t$hL\211\347\203\356\10\350\0\0\0\0HcCD\213SHH\215\4\200"..., 16384) = 16384
 1206  read(7, "H\211\312E\17\266\204$Z\4\0\0H\213{(M\215\264$:\4\0\0A\271\22\0\0\0\211\4"..., 32768) = 32768
 1206  read(7, "\17\266E@\307E\304\6\0\0\0\210E\325H\215E\320H\211E\270\351\305\376\377\377\17\37D\0\0"..., 65536) = 65536
-1206  mremap(0x7f9039f7f000, 135168, 266240, MREMAP_MAYMOVE) = 0x7f9039ac6000
+1206  mremap(0x7fed01062000, 135168, 266240, MREMAP_MAYMOVE) = 0x7fed00ba9000
 1206  read(7, "\32\230\0\0\0\0\0\0\2\0\0\0\224\2\0\0\374\377\377\377\377\377\377\377\"\230\0\0\0\0\0\0"..., 131072) = 122760
 1206  read(7, "", 8192)                 = 0
 1206  read(7, "", 8192)                 = 0
@@ -180826,770213 +257381,541898 @@
 1206  read(8, "", 4096)                 = 0
 1206  read(8, "", 4096)                 = 0
 1206  close(8)                          = 0
-1206  open("/lib/modules/2.6.32-220.2.1.el6.x86_64/kernel/net/ipv4/netfilter/arptable_filter.ko", O_RDONLY) = 8
-1206  fcntl(8, F_GETFL)                 = 0x8000 (flags O_RDONLY|O_LARGEFILE)
-1206  fstat(8, {st_mode=S_IFREG|0744, st_size=7848, ...}) = 0
-1206  lseek(8, 0, SEEK_CUR)             = 0
-1206  read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 16384) = 7848
-1206  read(8, "", 8192)                 = 0
-1206  read(8, "", 8192)                 = 0
-1206  read(8, "", 8192)                 = 0

[... a ton of the "-" lines, from yum localinstall ...]

-1206  write(3, "ss\240\3\306\276", 6)   = 6
-1206  lseek(3, 8, SEEK_SET)             = 8
-1206  write(3, " \3\307\233", 4)        = 4
-1206  lseek(3, 247713, SEEK_SET)        = 247713
-1206  close(3)                          = 0
-1206  rename("/lib/modules/2.6.32-220.2.1.el6.x86_64/modules.symbols.bin.temp", "/lib/modules/2.6.32-220.2.1.el6.x86_64/modules.symbols.bin
") = 0
-1206  exit_group(0)                     = ?
-1186  <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1206
+1206  +++ killed by SIGKILL +++
+1186  <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 8620
+1186  open("/usr/share/locale/locale.alias", O_RDONLY) = 3
+1186  fstat(3, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
+1186  read(3, "# Locale name alias data base.\n#"..., 4096) = 2512
+1186  read(3, "", 4096)                 = 0
+1186  close(3)                          = 0
+1186  open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en_US.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en_US/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  fstat(2, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0
+1186  open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+1186  write(2, "/sbin/new-kernel-pkg: line 309: "..., 111) = 111
 1186  --- SIGCHLD (Child exited) @ 0 (0) ---
-1186  wait4(-1, 0x7fff0a95d0dc, WNOHANG, NULL) = -1 ECHILD (No child processes)
+1186  wait4(-1, 0x7fff0c442adc, WNOHANG, NULL) = -1 ECHILD (No child processes)
 1186  rt_sigreturn(0xffffffffffffffff)  = 0
 1186  read(255, "\nif [ \"$initrd\" == \"make\" ]; the"..., 8192) = 749
 1186  lseek(255, -649, SEEK_CUR)        = 13169

Sadly, the error message written to the stderr is never shown anywhere, due to bug 606239, discovered by our fellow colleague Šimon.

So, the depmod -ae is killed under yum upgrade and yum install and thus the modules never get to the ramdisk. It does not get killed under yum localinstall and rpm -i.

Now: how do we go about figuring out what killed that depmod and why?

The full strace leading to that kill is:

9752  mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe05e795000
9752  lseek(8, 0, SEEK_CUR)             = 0
9752  read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 16384) = 8864
9752  read(8, "", 4096)                 = 0
9752  read(8, "", 4096)                 = 0
9752  read(8, "", 4096)                 = 0
9752  close(8)                          = 0
9752  brk(0x194ae000 <unfinished ...>
9752  +++ killed by SIGKILL +++
9732  <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 9752
9732  open("/usr/share/locale/locale.alias", O_RDONLY) = 3
9732  fstat(3, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0

The EC2 micro instance has 613 MB of RAM and no swap which seems to be the problem. When I add 1GB of swap, yum upgrade kernel populates the ramdisk just fine:

# free
             total       used       free     shared    buffers     cached
Mem:        604944     291044     313900          0       9908     204480
-/+ buffers/cache:      76656     528288
Swap:       999992      15032     984960
# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l
398

So the problem is that if the machine is low on memory, installation of new kernel package can leave it with no modules in the ramdisk. However, the depmod failure is hidden by yum because it does not show error messages from %posttrans, plus the new kernel gets configured to be booted the next time (to be the new default if default=0).

I got the same problem after "yum update" with the "Installed: kernel-2.6.32-220.2.1.el6.x86_64"...

On the default boot, the message shown...

    kernel panic - not syncing: attempted to kill init
    panic occurred, switching back to text console

... and nothing I can do.  Then I try to use the "CentOS Linux (2.6.32-71.29.1.el6.x86_64)", and run the command as Harald Hoyer's suggestion:

    # depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64 2.6.32-220.2.1.el6.x86_64
    # dracut --force '' 2.6.32-220.2.1.el6.x86_64

however, the problem cannot be resolved, and still show the msg...

    kernel panic - not syncing: attempted to kill init
    panic occurred, switching back to text console

I am not sure what can I do now, or reinstall the server? And DON'T update other server?

Thanks.

(In reply to comment #17)
> I got the same problem after "yum update" with the "Installed:
> kernel-2.6.32-220.2.1.el6.x86_64"...
> 
> On the default boot, the message shown...
> 
>     kernel panic - not syncing: attempted to kill init
>     panic occurred, switching back to text console
> 
> ... and nothing I can do.  Then I try to use the "CentOS Linux
> (2.6.32-71.29.1.el6.x86_64)", and run the command as Harald Hoyer's suggestion:
> 
>     # depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64
> 2.6.32-220.2.1.el6.x86_64
>     # dracut --force '' 2.6.32-220.2.1.el6.x86_64
> 
> however, the problem cannot be resolved, and still show the msg...
> 
>     kernel panic - not syncing: attempted to kill init
>     panic occurred, switching back to text console
> 
> I am not sure what can I do now, or reinstall the server? And DON'T update
> other server?
> 
> Thanks.

can you show me the output of:

# ls -l /boot

and your /boot/grub/grub.conf

Hi Hoyer

Thank you for your help.  Here are the info for your reference.  

Thank you again.

# ls -l /boot
total 66208
-rw-r--r--  1 root root   100947 Dec 23 17:14 config-2.6.32-220.2.1.el6.x86_64
-rw-r--r--  1 root root    97911 Jun 28  2011 config-2.6.32-71.29.1.el6.x86_64
-rw-r--r--. 1 root root    97862 May 20  2011 config-2.6.32-71.el6.x86_64
drwxr-xr-x. 3 root root     1024 Nov 28 17:38 efi
drwxr-xr-x. 2 root root     1024 Jan 11 11:36 grub
-rw-r--r--  1 root root 15846128 Jan 11 11:44 initramfs-2.6.32-220.2.1.el6.x86_64.img
-rw-r--r--  1 root root 13786323 Nov 28 18:27 initramfs-2.6.32-71.29.1.el6.x86_64.img
-rw-r--r--. 1 root root 13782620 Nov 28 17:40 initramfs-2.6.32-71.el6.x86_64.img
-rw-------  1 root root  5266920 Jan 11 10:47 initrd-2.6.32-71.29.1.el6.x86_64kdump.img
drwx------. 2 root root    12288 Nov 28 17:17 lost+found
-rw-r--r--  1 root root   171175 Dec 23 17:17 symvers-2.6.32-220.2.1.el6.x86_64.gz
-rw-r--r--  1 root root   160602 Jun 28  2011 symvers-2.6.32-71.29.1.el6.x86_64.gz
-rw-r--r--. 1 root root   160542 May 20  2011 symvers-2.6.32-71.el6.x86_64.gz
-rw-r--r--  1 root root  2313220 Dec 23 17:14 System.map-2.6.32-220.2.1.el6.x86_64
-rw-r--r--  1 root root  2228188 Jun 28  2011 System.map-2.6.32-71.29.1.el6.x86_64
-rw-r--r--. 1 root root  2226490 May 20  2011 System.map-2.6.32-71.el6.x86_64
-rwxr-xr-x  1 root root  3940752 Dec 23 17:14 vmlinuz-2.6.32-220.2.1.el6.x86_64
-rwxr-xr-x  1 root root  3795744 Jun 28  2011 vmlinuz-2.6.32-71.29.1.el6.x86_64
-rwxr-xr-x. 1 root root  3791040 May 20  2011 vmlinuz-2.6.32-71.el6.x86_64


# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/md127
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-220.2.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img
title CentOS Linux (2.6.32-71.29.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img
title CentOS (2.6.32-71.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-71.el6.x86_64.img

(In reply to comment #19)
> Hi Hoyer
> 
> Thank you for your help.  Here are the info for your reference.  
> 
> Thank you again.
> 
> # ls -l /boot
> total 66208
> -rw-r--r--  1 root root   100947 Dec 23 17:14 config-2.6.32-220.2.1.el6.x86_64
> -rw-r--r--  1 root root    97911 Jun 28  2011 config-2.6.32-71.29.1.el6.x86_64
> -rw-r--r--. 1 root root    97862 May 20  2011 config-2.6.32-71.el6.x86_64
> drwxr-xr-x. 3 root root     1024 Nov 28 17:38 efi
> drwxr-xr-x. 2 root root     1024 Jan 11 11:36 grub
> -rw-r--r--  1 root root 15846128 Jan 11 11:44
> initramfs-2.6.32-220.2.1.el6.x86_64.img
> -rw-r--r--  1 root root 13786323 Nov 28 18:27
> initramfs-2.6.32-71.29.1.el6.x86_64.img
> -rw-r--r--. 1 root root 13782620 Nov 28 17:40
> initramfs-2.6.32-71.el6.x86_64.img
> -rw-------  1 root root  5266920 Jan 11 10:47
> initrd-2.6.32-71.29.1.el6.x86_64kdump.img
> drwx------. 2 root root    12288 Nov 28 17:17 lost+found
> -rw-r--r--  1 root root   171175 Dec 23 17:17
> symvers-2.6.32-220.2.1.el6.x86_64.gz
> -rw-r--r--  1 root root   160602 Jun 28  2011
> symvers-2.6.32-71.29.1.el6.x86_64.gz
> -rw-r--r--. 1 root root   160542 May 20  2011 symvers-2.6.32-71.el6.x86_64.gz
> -rw-r--r--  1 root root  2313220 Dec 23 17:14
> System.map-2.6.32-220.2.1.el6.x86_64
> -rw-r--r--  1 root root  2228188 Jun 28  2011
> System.map-2.6.32-71.29.1.el6.x86_64
> -rw-r--r--. 1 root root  2226490 May 20  2011 System.map-2.6.32-71.el6.x86_64
> -rwxr-xr-x  1 root root  3940752 Dec 23 17:14 vmlinuz-2.6.32-220.2.1.el6.x86_64
> -rwxr-xr-x  1 root root  3795744 Jun 28  2011 vmlinuz-2.6.32-71.29.1.el6.x86_64
> -rwxr-xr-x. 1 root root  3791040 May 20  2011 vmlinuz-2.6.32-71.el6.x86_64
> 
> 
> # cat /boot/grub/grub.conf
> # grub.conf generated by anaconda
> #
> # Note that you do not have to rerun grub after making changes to this file
> # NOTICE:  You have a /boot partition.  This means that
> #          all kernel and initrd paths are relative to /boot/, eg.
> #          root (hd0,0)
> #          kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root
> #          initrd /initrd-[generic-]version.img
> #boot=/dev/md127
> default=0
> timeout=5
> splashimage=(hd0,0)/grub/splash.xpm.gz
> hiddenmenu
> title CentOS (2.6.32-220.2.1.el6.x86_64)
>         root (hd0,0)
>         kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64
> root=/dev/mapper/vg_dms-lv_root ro
> rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087
> rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root
> rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8
> SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb
> quiet
>         initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img
> title CentOS Linux (2.6.32-71.29.1.el6.x86_64)
>         root (hd0,0)
>         kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro
> root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087
> rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root
> rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8
> SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb
> quiet
>         initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img
> title CentOS (2.6.32-71.el6.x86_64)
>         root (hd0,0)
>         kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root
> rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087
> rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root
> rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8
> SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb
> quiet
>         initrd /initramfs-2.6.32-71.el6.x86_64.img

try to add: "rd_NO_MDADMCONF"

if you have console access you can add "rdshell rdinfo rdinitdebug" and you might get more info on the console

(In reply to comment #20)

> 
> try to add: "rd_NO_MDADMCONF"
> 
> if you have console access you can add "rdshell rdinfo rdinitdebug" and you
> might get more info on the console

I have the console access, but can you tell me where I add the "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"?

Thank you for your help.

(In reply to comment #21)
> (In reply to comment #20)
> 
> > 
> > try to add: "rd_NO_MDADMCONF"
> > 
> > if you have console access you can add "rdshell rdinfo rdinitdebug" and you
> > might get more info on the console
> 
> I have the console access, but can you tell me where I add the
> "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"?
> 
> Thank you for your help.

/etc/grub.conf at the end of the "append" line

(In reply to comment #22)
> (In reply to comment #21)
> > (In reply to comment #20)
> > 
> > > 
> > > try to add: "rd_NO_MDADMCONF"
> > > 
> > > if you have console access you can add "rdshell rdinfo rdinitdebug" and you
> > > might get more info on the console
> > 
> > I have the console access, but can you tell me where I add the
> > "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"?
> > 
> > Thank you for your help.
> 
> /etc/grub.conf at the end of the "append" line

Thank you for the help.

Append the "rdshell rdinfo rdinitdebug" at the end of /etc/grub.conf, for the "normal" boot, the msg is same:

    kernel panic - not syncing: attempted to kill init
    panic occurred, switching back to text console

Or, anything misunderstanding on the "append"?

[root@server]# cat /boot/grub/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root
#          initrd /initrd-[generic-]version.img
#boot=/dev/md127
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-220.2.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img
title CentOS Linux (2.6.32-71.29.1.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img
title CentOS (2.6.32-71.el6.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet
        initrd /initramfs-2.6.32-71.el6.x86_64.img
rdshell rdinfo rdinitdebug

> title CentOS (2.6.32-220.2.1.el6.x86_64)
>         root (hd0,0)
>         kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64
> root=/dev/mapper/vg_dms-lv_root ro
> rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087
> rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root
> rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8
> SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb
> quiet
>         initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img


append in the line starting with "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64" after "quiet" ... not at the end of the file.

1. remove "rhgb" and add "rdshell rdinfo rdinitdebug rd_NO_MDADMCONF" and reboot in the first kernel


2. if that does not work try to remove everything from the line starting with "kernel" so that it reads:

kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro

(In reply to comment #24)
> > title CentOS (2.6.32-220.2.1.el6.x86_64)
> >         root (hd0,0)
> >         kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64
> > root=/dev/mapper/vg_dms-lv_root ro
> > rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087
> > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root
> > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8
> > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb
> > quiet
> >         initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img
> 
> 
> append in the line starting with "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64"
> after "quiet" ... not at the end of the file.
> 
> 1. remove "rhgb" and add "rdshell rdinfo rdinitdebug rd_NO_MDADMCONF" and
> reboot in the first kernel
> 
> 
> 2. if that does not work try to remove everything from the line starting with
> "kernel" so that it reads:
> 
> kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro

Sorry for my misunderstanding...

1. Edit the line from

    kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet

to

kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M quiet rdshell rdinfo rdinitdebug rd_NO_MDADMCONF

there is a lot of script dump to screen, and "dracut:/#" shown finally. I'm not sure what should I dump for debug... =(

(start)
--------------------------------------------------------------------------------
...
...
...
+ export PS1=dracut:${PWD}#
+ [ -e /.profile ]
+ echo exec 0<>/dev/console 1<>/dev/console 2<>/dev/console
+ sh -i -l
sh: can't access tty; job control turned off
dracut:/#
--------------------------------------------------------------------------------
(end)




2. "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro", and the finally screen as below:

(begin)
--------------------------------------------------------------------------------
...
...
...
firewire_coreL created device fw0: GUID 001e8c0000f36025, S400
dracut Warning: No root device "block:/dev/mapper/vg_XXX-lv_root" found




dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.


dracut Warning: Signal caught!

dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line.
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-220.2.1.el6.x86_64 #1
Call Trace:
 [<ffffffff814ec3b9>] ? panic+0x78/0x143
 [<ffffffff8106ecf2>] ? do_exit+0x852/0x860
 [<ffffffff81177de5>] ? fput+0x25/0x30
 [<ffffffff8106ed58>] ? do_group_exit+0x58/0xd0
 [<ffffffff8106ede7>] ? sys_exit_group+0x17/0x20
 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
panic occurred, switching back to text console
--------------------------------------------------------------------------------
(end)

(In reply to comment #17)
> 
> however, the problem cannot be resolved, and still show the msg...
> 
>     kernel panic - not syncing: attempted to kill init
>     panic occurred, switching back to text console
> 
> I am not sure what can I do now, or reinstall the server? And DON'T update
> other server?

Can you check if the kernel modules are in the ramdisk after you run the depmod? If they are, it's some different issue and it probably should be tracked in different bugzilla.

(In reply to comment #26)
> (In reply to comment #17)
> > 
> > however, the problem cannot be resolved, and still show the msg...
> > 
> >     kernel panic - not syncing: attempted to kill init
> >     panic occurred, switching back to text console
> > 
> > I am not sure what can I do now, or reinstall the server? And DON'T update
> > other server?
> 
> Can you check if the kernel modules are in the ramdisk after you run the
> depmod? If they are, it's some different issue and it probably should be
> tracked in different bugzilla.

Could you tell me how to check?  Sorry that I just an entry level user in the Linux World.

(In reply to comment #27)
> 
> Could you tell me how to check?  Sorry that I just an entry level user in the
> Linux World.

# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l

is what I've been using (comment 13).

(In reply to comment #28)
> (In reply to comment #27)
> > 
> > Could you tell me how to check?  Sorry that I just an entry level user in the
> > Linux World.
> 
> # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules |
> wc -l
> 
> is what I've been using (comment 13).

Hi, 

The output only "412"

[root@server ~]# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l
412
[root@server ~]#

(In reply to comment #29)
> 
> The output only "412"
> 
> [root@server ~]# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep
> lib/modules | wc -l
> 412
> [root@server ~]#

Great. If the output is greater than zero, you have the modules, and your issue is different than the one tracked in this bugzilla. You might want to open a new one because chances are your problem is in completely different component.

I may be seeing this issue with some brand new KVM guests that I am installing with virt-install:

host ~ # losetup --offset 1048576 /dev/loop0 /dev/mapper/vg_host-el6--guest.img 
host ~ # mount -t ext4 -o ro /dev/loop0 /mnt
host ~ # lsinitrd /mnt/boot/initramfs-2.6.32-220.4.1.el6.x86_64.img | grep lib/modules | wc -l
0

When I boot with "rd_NO_MDADMCONF rdshell rdinfo rdinitdebug" I end up with this:

dracut: + udevadm settle --timeout=0
dracut: + modprobe scsi_wait_scan
dracut: FATAL: Could not load /lib/modules/2.6.32-220.4.1.el6.x86_64/modules.dep: No such file or directory

That isn't the same error message, but since the initramfs appears to lack modules am I seeing the same bug?  I can get to a debug shell, so I can inspect things from there if you wish.

(In reply to comment #31)
> /lib/modules/2.6.32-220.4.1.el6.x86_64/modules.dep: No such file or directory
> 

depmod did not run -> same bug

The latest kernel update 2.6.32-220.7.1 doesn't work with Amazon EC2.

Can you show the yum output from the part where the packages are actually being installed?

Created attachment 578073 [details]
Output of yum upgrade -v kernel

Please find the log attached.

Alright, we're going to need to do some more debugging then.  Can you remove that kernel package, then install it with:

rpm -i --noscripts kernel.x86_64 0:2.6.32-220.7.1.el6.x86_64.rpm

and then run: 

sh -x /sbin/new-kernel-pkg --package kernel --install 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1
sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-220.7.1.el6.x86_64 crashkernel=128M > mkinitrd.log 2>&1

And then attach install.log and mkinitrd.log here?

(In reply to comment #39)
> Alright, we're going to need to do some more debugging then.  Can you remove
> that kernel package, then install it with:
> 
> rpm -i --noscripts kernel.x86_64 0:2.6.32-220.7.1.el6.x86_64.rpm

That should be:

"rpm -i noscripts kernel-$WHATEVER.rpm"

for whatever your correct kernel version is, of course.  And make sure the version matches in the other commands as well.

Also I got the second command (generating mkinitrd.log) slightly wrong.  Please run:

sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1

Created attachment 579441 [details]
The install.log

# rpm -ivh --noscripts kernel-2.6.32-220.7.1.el6.x86_64.rpm kernel-firmware-2.6.32-220.7.1.el6.noarch.rpm 
Preparing...                ########################################### [100%]
   1:kernel-firmware        ########################################### [ 50%]
   2:kernel                 ########################################### [100%]
# sh -x /sbin/new-kernel-pkg --package kernel --install 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1
# sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1

Created attachment 579444 [details]
The mkinitrd.log

(In reply to comment #42)
> Created attachment 579441 [details]
> The install.log
> 
> # rpm -ivh --noscripts kernel-2.6.32-220.7.1.el6.x86_64.rpm
> kernel-firmware-2.6.32-220.7.1.el6.noarch.rpm 
> Preparing...                ########################################### [100%]
>    1:kernel-firmware        ########################################### [ 50%]
>    2:kernel                 ########################################### [100%]
> # sh -x /sbin/new-kernel-pkg --package kernel --install
> 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1
> # sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod
> --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log
> 2>&1

last command has the wrong kernel version!!

Created attachment 579493 [details]
The mkinitrd.log

Retried with

# sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-220.7.1.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1

From that log:

+ doDepmod
+ '[' -n '' ']'
+ depmod -ae -F /boot/System.map-2.6.32-220.7.1.el6.x86_64 2.6.32-220.7.1.el6.x86_64
+ '[' make == make ']'
+ mkinitrd
+ '[' -n --dracut ']'
+ tool='/sbin/dracut  -f /boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 2.6.32-220.7.1.el6.x86_64'

As per that, it appears that we're running depmod.  If its results aren't there, unless there's some reason to believe otherwise, it seems that this is a depmod bug.

Hi,

depmod was leaking memory, the fix in https://bugzilla.redhat.com/show_bug.cgi?id=670613 should make it work again.

Do you think it might be a DUP of 670613? Can you test with 6.3 nightlies somehow?

(In reply to comment #48)
> Hi,
> 
> depmod was leaking memory, the fix in
> https://bugzilla.redhat.com/show_bug.cgi?id=670613 should make it work again.
> 
> Do you think it might be a DUP of 670613? Can you test with 6.3 nightlies
> somehow?

I don't see RHEL 6.3 available in EC2. If you were able to reproduce the original issue of this bugzilla on some small memory non-EC2, you should be able to check 6.3 as well.

I confirm that where

module-init-tools-3.9-17.el6.x86_64

produces

# for i in /boot/initramfs*.img ; do echo -n "$i " ; lsinitrd $i | grep lib/modules | wc -l ; done
/boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 0
/boot/initramfs-2.6.32-220.el6.x86_64.img 398

after yum upgrade, with

module-init-tools-3.9-19.el6_2.x86_64

the bug is fixed:

# for i in /boot/initramfs*.img ; do echo -n "$i " ; lsinitrd $i | grep lib/modules | wc -l ; done
/boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 398
/boot/initramfs-2.6.32-220.el6.x86_64.img 398

Feel free to close this bug as duplicate of bug 670613, even if you might want to keep this bugzilla open and specifically tested in the EC2 environment.

*** This bug has been marked as a duplicate of bug 670613 ***