Hide Forgot
Description of problem: I've started new micro 64bit instance on EC2, with AMI: RHEL-6.2-Starter-EBS-x86_64-4-Access2 (ami-4fd00526). It booted with kernel 2.6.32-220.el6.x86_64. Running yum upgrade showed upgrade of kernel to 2.6.32-220.2.1.el6.x86_64 which I did: Dependencies Resolved ================================================================================================================================================================ Package Arch Version Repository Size ================================================================================================================================================================ Installing: kernel x86_64 2.6.32-220.2.1.el6 rhui-us-east-1-rhel-server-releases 24 M Updating for dependencies: kernel-firmware noarch 2.6.32-220.2.1.el6 rhui-us-east-1-rhel-server-releases 6.2 M Transaction Summary ================================================================================================================================================================ Install 1 Package(s) Upgrade 1 Package(s) Total download size: 31 M After restarting the instance (stop and start), it never comes up and ec2-get-console-output shows NET: Registered protocol family 17 registered taskstats version 1 XENBUS: Device with no driver: device/vbd/2049 XENBUS: Device with no driver: device/vif/0 XENBUS: Device with no driver: device/console/0 drivers/rtc/hctosys.c: unable to open rtc device (rtc0) Initalizing network drop monitor service Freeing unused kernel memory: 1244k freed Write protecting the kernel read-only data: 10240k Freeing unused kernel memory: 1040k freed Freeing unused kernel memory: 1760k freed dracut: dracut-004-256.el6 udev: starting version 147 dracut: Starting plymouth daemon dracut Warning: No root device "block:/dev/disk/by-label/_\x2f" found dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. dracut Warning: Signal caught! dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. Kernel panic - not syncing: Attempted to kill init! Pid: 1, comm: init Not tainted 2.6.32-220.2.1.el6.x86_64 #1 Call Trace: [<ffffffff814ec3b9>] ? panic+0x78/0x143 [<ffffffff8106ecf2>] ? do_exit+0x852/0x860 [<ffffffff81177de5>] ? fput+0x25/0x30 [<ffffffff8106ed58>] ? do_group_exit+0x58/0xd0 [<ffffffff8106ede7>] ? sys_exit_group+0x17/0x20 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b Version-Release number of selected component (if applicable): kernel-2.6.32-220.2.1.el6.x86_64 dracut-004-256.el6.noarch How reproducible: Deterministic. Steps to Reproduce: 1. Upgrade EC2 micro instance to latest kernel, it will fail to upgrade afterwards. Actual results: dracut: dracut-004-256.el6 udev: starting version 147 dracut: Starting plymouth daemon dracut Warning: No root device "block:/dev/disk/by-label/_\x2f" found dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. dracut Warning: Signal caught! dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. Kernel panic - not syncing: Attempted to kill init! Pid: 1, comm: init Not tainted 2.6.32-220.2.1.el6.x86_64 #1 Call Trace: [<ffffffff814ec3b9>] ? panic+0x78/0x143 [<ffffffff8106ecf2>] ? do_exit+0x852/0x860 [<ffffffff81177de5>] ? fput+0x25/0x30 [<ffffffff8106ed58>] ? do_group_exit+0x58/0xd0 [<ffffffff8106ede7>] ? sys_exit_group+0x17/0x20 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b after reboot. Expected results: No error after reboot. Additional info: There are multiple people reporting the same problem: https://forums.aws.amazon.com/thread.jspa?threadID=78101 https://forums.aws.amazon.com/thread.jspa?threadID=81632 https://forums.aws.amazon.com/thread.jspa?threadID=72610 The workaround is to change the default in /boot/grub/grub.conf to point to the old version of the kernel. I am not sure if this is a bug in dracut, kernel, or RHUI, so please reassign as needed.
what is the kernel command line, please show /boot/grub/grub.conf
# cat /boot/grub/grub.conf default=1 timeout=0 hiddenmenu title Red Hat Enterprise Linux Server (2.6.32-220.2.1.el6.x86_64) root (hd0) kernel /boot/vmlinuz-2.6.32-220.2.1.el6.x86_64 ro root=LABEL=_/ crashkernel=auto LANG=en_US.UTF-8 KEYTABLE=us initrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img title koji-image-RHEL-6.2-build-3896128-201112151048 (2.6.32-220.el6.x86_64) root (hd0) kernel /boot/vmlinuz-2.6.32-220.el6.x86_64 ro root=LABEL=_/ initrd /boot/initramfs-2.6.32-220.el6.x86_64.img
Umm, re-reading my Steps to reproduce, it needs the s/it will fail to upgrade afterwards/it will fail to boot afterwards change.
can you attach the output of: # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img and # lsinitrd /boot/initramfs-2.6.32-220.el6.x86_64.img
Created attachment 550899 [details] lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img
Created attachment 550901 [details] lsinitrd /boot/initramfs-2.6.32-220.el6.x86_64.img
Fresh install... kernel-2.6.32-220.el6.x86_64 was running I did a # yum install kernel so, I see that depmod was not run (or failed??) before dracut was called in the kernel %posttrans /lib/modules/2.6.32-220.2.1.el6.x86_64/modules.dep does not exists this points to /sbin/new-kernel-pkg
after doing # depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64 2.6.32-220.2.1.el6.x86_64 # dracut --force '' 2.6.32-220.2.1.el6.x86_64 all was fine after a reboot
I see the same problems for F16 sometime. Missing depmod, missing initramfs line, etc..
I've tried to upgrade the kernel a couple more times, with the following results: - SELinux does not play any role; - rpm -i and yum localinstall place the modules to the ramdisk just fine; - yum upgrade and yum install do not place the modules to the ramdisk. I was testing with # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l after the upgrade / installation, then did # rpm -e kernel-2.6.32-220.2.1.el6.x86_64 and started anew with the next type of upgrade. Things are deterministic.
I've run yum localinstall kernel-2.6.32-220.2.1.el6.x86_64.rpm and yum upgrade kernel under strace -f and started diffing the log, after (manually) modifying the pids in the log of respective processes to match. The difference is at 1206 execve("/sbin/depmod", ["depmod", "-ae", "-F", "/boot/System.map-2.6.32-220.2.1."..., "2.6.32-220.2.1.el6.x86_64"], [/* 27 vars */]) = 0 @@ -160548,7 +237103,7 @@ 1206 read(7, "\213O\330H\201\301\260\0\0\0\351\r\377\377\377\220D\213K\330A\201\301\260\0\0\0\351\21\377\377\377"..., 16384) = 16384 1206 read(7, "\0\0\0\0L\213f\20I\211\365L\215u\200L\215}\260H\211\373\350\0\0\0\0I\215u0L"..., 32768) = 32768 [...] 1206 read(7, "\351'\377\377\377A\213t$hL\211\347\203\356\10\350\0\0\0\0HcCD\213SHH\215\4\200"..., 16384) = 16384 1206 read(7, "H\211\312E\17\266\204$Z\4\0\0H\213{(M\215\264$:\4\0\0A\271\22\0\0\0\211\4"..., 32768) = 32768 1206 read(7, "\17\266E@\307E\304\6\0\0\0\210E\325H\215E\320H\211E\270\351\305\376\377\377\17\37D\0\0"..., 65536) = 65536 -1206 mremap(0x7f9039f7f000, 135168, 266240, MREMAP_MAYMOVE) = 0x7f9039ac6000 +1206 mremap(0x7fed01062000, 135168, 266240, MREMAP_MAYMOVE) = 0x7fed00ba9000 1206 read(7, "\32\230\0\0\0\0\0\0\2\0\0\0\224\2\0\0\374\377\377\377\377\377\377\377\"\230\0\0\0\0\0\0"..., 131072) = 122760 1206 read(7, "", 8192) = 0 1206 read(7, "", 8192) = 0 @@ -180826,770213 +257381,541898 @@ 1206 read(8, "", 4096) = 0 1206 read(8, "", 4096) = 0 1206 close(8) = 0 -1206 open("/lib/modules/2.6.32-220.2.1.el6.x86_64/kernel/net/ipv4/netfilter/arptable_filter.ko", O_RDONLY) = 8 -1206 fcntl(8, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) -1206 fstat(8, {st_mode=S_IFREG|0744, st_size=7848, ...}) = 0 -1206 lseek(8, 0, SEEK_CUR) = 0 -1206 read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 16384) = 7848 -1206 read(8, "", 8192) = 0 -1206 read(8, "", 8192) = 0 -1206 read(8, "", 8192) = 0 [... a ton of the "-" lines, from yum localinstall ...] -1206 write(3, "ss\240\3\306\276", 6) = 6 -1206 lseek(3, 8, SEEK_SET) = 8 -1206 write(3, " \3\307\233", 4) = 4 -1206 lseek(3, 247713, SEEK_SET) = 247713 -1206 close(3) = 0 -1206 rename("/lib/modules/2.6.32-220.2.1.el6.x86_64/modules.symbols.bin.temp", "/lib/modules/2.6.32-220.2.1.el6.x86_64/modules.symbols.bin ") = 0 -1206 exit_group(0) = ? -1186 <... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1206 +1206 +++ killed by SIGKILL +++ +1186 <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 8620 +1186 open("/usr/share/locale/locale.alias", O_RDONLY) = 3 +1186 fstat(3, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0 +1186 read(3, "# Locale name alias data base.\n#"..., 4096) = 2512 +1186 read(3, "", 4096) = 0 +1186 close(3) = 0 +1186 open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en_US.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en_US/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en.UTF-8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en.utf8/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en/LC_MESSAGES/bash.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 fstat(2, {st_mode=S_IFREG|0600, st_size=0, ...}) = 0 +1186 open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory) +1186 write(2, "/sbin/new-kernel-pkg: line 309: "..., 111) = 111 1186 --- SIGCHLD (Child exited) @ 0 (0) --- -1186 wait4(-1, 0x7fff0a95d0dc, WNOHANG, NULL) = -1 ECHILD (No child processes) +1186 wait4(-1, 0x7fff0c442adc, WNOHANG, NULL) = -1 ECHILD (No child processes) 1186 rt_sigreturn(0xffffffffffffffff) = 0 1186 read(255, "\nif [ \"$initrd\" == \"make\" ]; the"..., 8192) = 749 1186 lseek(255, -649, SEEK_CUR) = 13169 Sadly, the error message written to the stderr is never shown anywhere, due to bug 606239, discovered by our fellow colleague Šimon.
So, the depmod -ae is killed under yum upgrade and yum install and thus the modules never get to the ramdisk. It does not get killed under yum localinstall and rpm -i. Now: how do we go about figuring out what killed that depmod and why?
The full strace leading to that kill is: 9752 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe05e795000 9752 lseek(8, 0, SEEK_CUR) = 0 9752 read(8, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 16384) = 8864 9752 read(8, "", 4096) = 0 9752 read(8, "", 4096) = 0 9752 read(8, "", 4096) = 0 9752 close(8) = 0 9752 brk(0x194ae000 <unfinished ...> 9752 +++ killed by SIGKILL +++ 9732 <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], 0, NULL) = 9752 9732 open("/usr/share/locale/locale.alias", O_RDONLY) = 3 9732 fstat(3, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0 The EC2 micro instance has 613 MB of RAM and no swap which seems to be the problem. When I add 1GB of swap, yum upgrade kernel populates the ramdisk just fine: # free total used free shared buffers cached Mem: 604944 291044 313900 0 9908 204480 -/+ buffers/cache: 76656 528288 Swap: 999992 15032 984960 # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l 398 So the problem is that if the machine is low on memory, installation of new kernel package can leave it with no modules in the ramdisk. However, the depmod failure is hidden by yum because it does not show error messages from %posttrans, plus the new kernel gets configured to be booted the next time (to be the new default if default=0).
I got the same problem after "yum update" with the "Installed: kernel-2.6.32-220.2.1.el6.x86_64"... On the default boot, the message shown... kernel panic - not syncing: attempted to kill init panic occurred, switching back to text console ... and nothing I can do. Then I try to use the "CentOS Linux (2.6.32-71.29.1.el6.x86_64)", and run the command as Harald Hoyer's suggestion: # depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64 2.6.32-220.2.1.el6.x86_64 # dracut --force '' 2.6.32-220.2.1.el6.x86_64 however, the problem cannot be resolved, and still show the msg... kernel panic - not syncing: attempted to kill init panic occurred, switching back to text console I am not sure what can I do now, or reinstall the server? And DON'T update other server? Thanks.
(In reply to comment #17) > I got the same problem after "yum update" with the "Installed: > kernel-2.6.32-220.2.1.el6.x86_64"... > > On the default boot, the message shown... > > kernel panic - not syncing: attempted to kill init > panic occurred, switching back to text console > > ... and nothing I can do. Then I try to use the "CentOS Linux > (2.6.32-71.29.1.el6.x86_64)", and run the command as Harald Hoyer's suggestion: > > # depmod -ae -F /boot/System.map-2.6.32-220.2.1.el6.x86_64 > 2.6.32-220.2.1.el6.x86_64 > # dracut --force '' 2.6.32-220.2.1.el6.x86_64 > > however, the problem cannot be resolved, and still show the msg... > > kernel panic - not syncing: attempted to kill init > panic occurred, switching back to text console > > I am not sure what can I do now, or reinstall the server? And DON'T update > other server? > > Thanks. can you show me the output of: # ls -l /boot and your /boot/grub/grub.conf
Hi Hoyer Thank you for your help. Here are the info for your reference. Thank you again. # ls -l /boot total 66208 -rw-r--r-- 1 root root 100947 Dec 23 17:14 config-2.6.32-220.2.1.el6.x86_64 -rw-r--r-- 1 root root 97911 Jun 28 2011 config-2.6.32-71.29.1.el6.x86_64 -rw-r--r--. 1 root root 97862 May 20 2011 config-2.6.32-71.el6.x86_64 drwxr-xr-x. 3 root root 1024 Nov 28 17:38 efi drwxr-xr-x. 2 root root 1024 Jan 11 11:36 grub -rw-r--r-- 1 root root 15846128 Jan 11 11:44 initramfs-2.6.32-220.2.1.el6.x86_64.img -rw-r--r-- 1 root root 13786323 Nov 28 18:27 initramfs-2.6.32-71.29.1.el6.x86_64.img -rw-r--r--. 1 root root 13782620 Nov 28 17:40 initramfs-2.6.32-71.el6.x86_64.img -rw------- 1 root root 5266920 Jan 11 10:47 initrd-2.6.32-71.29.1.el6.x86_64kdump.img drwx------. 2 root root 12288 Nov 28 17:17 lost+found -rw-r--r-- 1 root root 171175 Dec 23 17:17 symvers-2.6.32-220.2.1.el6.x86_64.gz -rw-r--r-- 1 root root 160602 Jun 28 2011 symvers-2.6.32-71.29.1.el6.x86_64.gz -rw-r--r--. 1 root root 160542 May 20 2011 symvers-2.6.32-71.el6.x86_64.gz -rw-r--r-- 1 root root 2313220 Dec 23 17:14 System.map-2.6.32-220.2.1.el6.x86_64 -rw-r--r-- 1 root root 2228188 Jun 28 2011 System.map-2.6.32-71.29.1.el6.x86_64 -rw-r--r--. 1 root root 2226490 May 20 2011 System.map-2.6.32-71.el6.x86_64 -rwxr-xr-x 1 root root 3940752 Dec 23 17:14 vmlinuz-2.6.32-220.2.1.el6.x86_64 -rwxr-xr-x 1 root root 3795744 Jun 28 2011 vmlinuz-2.6.32-71.29.1.el6.x86_64 -rwxr-xr-x. 1 root root 3791040 May 20 2011 vmlinuz-2.6.32-71.el6.x86_64 # cat /boot/grub/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root # initrd /initrd-[generic-]version.img #boot=/dev/md127 default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.32-220.2.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img title CentOS Linux (2.6.32-71.29.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img title CentOS (2.6.32-71.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-71.el6.x86_64.img
(In reply to comment #19) > Hi Hoyer > > Thank you for your help. Here are the info for your reference. > > Thank you again. > > # ls -l /boot > total 66208 > -rw-r--r-- 1 root root 100947 Dec 23 17:14 config-2.6.32-220.2.1.el6.x86_64 > -rw-r--r-- 1 root root 97911 Jun 28 2011 config-2.6.32-71.29.1.el6.x86_64 > -rw-r--r--. 1 root root 97862 May 20 2011 config-2.6.32-71.el6.x86_64 > drwxr-xr-x. 3 root root 1024 Nov 28 17:38 efi > drwxr-xr-x. 2 root root 1024 Jan 11 11:36 grub > -rw-r--r-- 1 root root 15846128 Jan 11 11:44 > initramfs-2.6.32-220.2.1.el6.x86_64.img > -rw-r--r-- 1 root root 13786323 Nov 28 18:27 > initramfs-2.6.32-71.29.1.el6.x86_64.img > -rw-r--r--. 1 root root 13782620 Nov 28 17:40 > initramfs-2.6.32-71.el6.x86_64.img > -rw------- 1 root root 5266920 Jan 11 10:47 > initrd-2.6.32-71.29.1.el6.x86_64kdump.img > drwx------. 2 root root 12288 Nov 28 17:17 lost+found > -rw-r--r-- 1 root root 171175 Dec 23 17:17 > symvers-2.6.32-220.2.1.el6.x86_64.gz > -rw-r--r-- 1 root root 160602 Jun 28 2011 > symvers-2.6.32-71.29.1.el6.x86_64.gz > -rw-r--r--. 1 root root 160542 May 20 2011 symvers-2.6.32-71.el6.x86_64.gz > -rw-r--r-- 1 root root 2313220 Dec 23 17:14 > System.map-2.6.32-220.2.1.el6.x86_64 > -rw-r--r-- 1 root root 2228188 Jun 28 2011 > System.map-2.6.32-71.29.1.el6.x86_64 > -rw-r--r--. 1 root root 2226490 May 20 2011 System.map-2.6.32-71.el6.x86_64 > -rwxr-xr-x 1 root root 3940752 Dec 23 17:14 vmlinuz-2.6.32-220.2.1.el6.x86_64 > -rwxr-xr-x 1 root root 3795744 Jun 28 2011 vmlinuz-2.6.32-71.29.1.el6.x86_64 > -rwxr-xr-x. 1 root root 3791040 May 20 2011 vmlinuz-2.6.32-71.el6.x86_64 > > > # cat /boot/grub/grub.conf > # grub.conf generated by anaconda > # > # Note that you do not have to rerun grub after making changes to this file > # NOTICE: You have a /boot partition. This means that > # all kernel and initrd paths are relative to /boot/, eg. > # root (hd0,0) > # kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root > # initrd /initrd-[generic-]version.img > #boot=/dev/md127 > default=0 > timeout=5 > splashimage=(hd0,0)/grub/splash.xpm.gz > hiddenmenu > title CentOS (2.6.32-220.2.1.el6.x86_64) > root (hd0,0) > kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 > root=/dev/mapper/vg_dms-lv_root ro > rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb > quiet > initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img > title CentOS Linux (2.6.32-71.29.1.el6.x86_64) > root (hd0,0) > kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro > root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb > quiet > initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img > title CentOS (2.6.32-71.el6.x86_64) > root (hd0,0) > kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root > rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb > quiet > initrd /initramfs-2.6.32-71.el6.x86_64.img try to add: "rd_NO_MDADMCONF" if you have console access you can add "rdshell rdinfo rdinitdebug" and you might get more info on the console
(In reply to comment #20) > > try to add: "rd_NO_MDADMCONF" > > if you have console access you can add "rdshell rdinfo rdinitdebug" and you > might get more info on the console I have the console access, but can you tell me where I add the "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"? Thank you for your help.
(In reply to comment #21) > (In reply to comment #20) > > > > > try to add: "rd_NO_MDADMCONF" > > > > if you have console access you can add "rdshell rdinfo rdinitdebug" and you > > might get more info on the console > > I have the console access, but can you tell me where I add the > "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"? > > Thank you for your help. /etc/grub.conf at the end of the "append" line
(In reply to comment #22) > (In reply to comment #21) > > (In reply to comment #20) > > > > > > > > try to add: "rd_NO_MDADMCONF" > > > > > > if you have console access you can add "rdshell rdinfo rdinitdebug" and you > > > might get more info on the console > > > > I have the console access, but can you tell me where I add the > > "rd_NO_MDADMCONF" and "rdshell rdinfo rdinitdebug"? > > > > Thank you for your help. > > /etc/grub.conf at the end of the "append" line Thank you for the help. Append the "rdshell rdinfo rdinitdebug" at the end of /etc/grub.conf, for the "normal" boot, the msg is same: kernel panic - not syncing: attempted to kill init panic occurred, switching back to text console Or, anything misunderstanding on the "append"? [root@server]# cat /boot/grub/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/mapper/vg_dms-lv_root # initrd /initrd-[generic-]version.img #boot=/dev/md127 default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.32-220.2.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img title CentOS Linux (2.6.32-71.29.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-71.29.1.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-71.29.1.el6.x86_64.img title CentOS (2.6.32-71.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_dms-lv_root rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet initrd /initramfs-2.6.32-71.el6.x86_64.img rdshell rdinfo rdinitdebug
> title CentOS (2.6.32-220.2.1.el6.x86_64) > root (hd0,0) > kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 > root=/dev/mapper/vg_dms-lv_root ro > rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb > quiet > initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img append in the line starting with "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64" after "quiet" ... not at the end of the file. 1. remove "rhgb" and add "rdshell rdinfo rdinitdebug rd_NO_MDADMCONF" and reboot in the first kernel 2. if that does not work try to remove everything from the line starting with "kernel" so that it reads: kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro
(In reply to comment #24) > > title CentOS (2.6.32-220.2.1.el6.x86_64) > > root (hd0,0) > > kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 > > root=/dev/mapper/vg_dms-lv_root ro > > rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 > > rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root > > rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 > > SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb > > quiet > > initrd /initramfs-2.6.32-220.2.1.el6.x86_64.img > > > append in the line starting with "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64" > after "quiet" ... not at the end of the file. > > 1. remove "rhgb" and add "rdshell rdinfo rdinitdebug rd_NO_MDADMCONF" and > reboot in the first kernel > > > 2. if that does not work try to remove everything from the line starting with > "kernel" so that it reads: > > kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro Sorry for my misunderstanding... 1. Edit the line from kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb quiet to kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro rd_MD_UUID=117d96cc:0c02e3f1:5fb9c8a8:aa31d087 rd_MD_UUID=b4eddb44:183ed058:ddd91bd7:8d6e7790 rd_LVM_LV=vg_dms/lv_root rd_LVM_LV=vg_dms/lv_swap rd_NO_LUKS rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M quiet rdshell rdinfo rdinitdebug rd_NO_MDADMCONF there is a lot of script dump to screen, and "dracut:/#" shown finally. I'm not sure what should I dump for debug... =( (start) -------------------------------------------------------------------------------- ... ... ... + export PS1=dracut:${PWD}# + [ -e /.profile ] + echo exec 0<>/dev/console 1<>/dev/console 2<>/dev/console + sh -i -l sh: can't access tty; job control turned off dracut:/# -------------------------------------------------------------------------------- (end) 2. "kernel /vmlinuz-2.6.32-220.2.1.el6.x86_64 root=/dev/mapper/vg_dms-lv_root ro", and the finally screen as below: (begin) -------------------------------------------------------------------------------- ... ... ... firewire_coreL created device fw0: GUID 001e8c0000f36025, S400 dracut Warning: No root device "block:/dev/mapper/vg_XXX-lv_root" found dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. dracut Warning: Signal caught! dracut Warning: Boot has failed. To debug this issue add "rdshell" to the kernel command line. Kernel panic - not syncing: Attempted to kill init! Pid: 1, comm: init Not tainted 2.6.32-220.2.1.el6.x86_64 #1 Call Trace: [<ffffffff814ec3b9>] ? panic+0x78/0x143 [<ffffffff8106ecf2>] ? do_exit+0x852/0x860 [<ffffffff81177de5>] ? fput+0x25/0x30 [<ffffffff8106ed58>] ? do_group_exit+0x58/0xd0 [<ffffffff8106ede7>] ? sys_exit_group+0x17/0x20 [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b panic occurred, switching back to text console -------------------------------------------------------------------------------- (end)
(In reply to comment #17) > > however, the problem cannot be resolved, and still show the msg... > > kernel panic - not syncing: attempted to kill init > panic occurred, switching back to text console > > I am not sure what can I do now, or reinstall the server? And DON'T update > other server? Can you check if the kernel modules are in the ramdisk after you run the depmod? If they are, it's some different issue and it probably should be tracked in different bugzilla.
(In reply to comment #26) > (In reply to comment #17) > > > > however, the problem cannot be resolved, and still show the msg... > > > > kernel panic - not syncing: attempted to kill init > > panic occurred, switching back to text console > > > > I am not sure what can I do now, or reinstall the server? And DON'T update > > other server? > > Can you check if the kernel modules are in the ramdisk after you run the > depmod? If they are, it's some different issue and it probably should be > tracked in different bugzilla. Could you tell me how to check? Sorry that I just an entry level user in the Linux World.
(In reply to comment #27) > > Could you tell me how to check? Sorry that I just an entry level user in the > Linux World. # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l is what I've been using (comment 13).
(In reply to comment #28) > (In reply to comment #27) > > > > Could you tell me how to check? Sorry that I just an entry level user in the > > Linux World. > > # lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | > wc -l > > is what I've been using (comment 13). Hi, The output only "412" [root@server ~]# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep lib/modules | wc -l 412 [root@server ~]#
(In reply to comment #29) > > The output only "412" > > [root@server ~]# lsinitrd /boot/initramfs-2.6.32-220.2.1.el6.x86_64.img | grep > lib/modules | wc -l > 412 > [root@server ~]# Great. If the output is greater than zero, you have the modules, and your issue is different than the one tracked in this bugzilla. You might want to open a new one because chances are your problem is in completely different component.
I may be seeing this issue with some brand new KVM guests that I am installing with virt-install: host ~ # losetup --offset 1048576 /dev/loop0 /dev/mapper/vg_host-el6--guest.img host ~ # mount -t ext4 -o ro /dev/loop0 /mnt host ~ # lsinitrd /mnt/boot/initramfs-2.6.32-220.4.1.el6.x86_64.img | grep lib/modules | wc -l 0 When I boot with "rd_NO_MDADMCONF rdshell rdinfo rdinitdebug" I end up with this: dracut: + udevadm settle --timeout=0 dracut: + modprobe scsi_wait_scan dracut: FATAL: Could not load /lib/modules/2.6.32-220.4.1.el6.x86_64/modules.dep: No such file or directory That isn't the same error message, but since the initramfs appears to lack modules am I seeing the same bug? I can get to a debug shell, so I can inspect things from there if you wish.
(In reply to comment #31) > /lib/modules/2.6.32-220.4.1.el6.x86_64/modules.dep: No such file or directory > depmod did not run -> same bug
The latest kernel update 2.6.32-220.7.1 doesn't work with Amazon EC2.
Can you show the yum output from the part where the packages are actually being installed?
Created attachment 578073 [details] Output of yum upgrade -v kernel Please find the log attached.
Alright, we're going to need to do some more debugging then. Can you remove that kernel package, then install it with: rpm -i --noscripts kernel.x86_64 0:2.6.32-220.7.1.el6.x86_64.rpm and then run: sh -x /sbin/new-kernel-pkg --package kernel --install 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1 sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-220.7.1.el6.x86_64 crashkernel=128M > mkinitrd.log 2>&1 And then attach install.log and mkinitrd.log here?
(In reply to comment #39) > Alright, we're going to need to do some more debugging then. Can you remove > that kernel package, then install it with: > > rpm -i --noscripts kernel.x86_64 0:2.6.32-220.7.1.el6.x86_64.rpm That should be: "rpm -i noscripts kernel-$WHATEVER.rpm" for whatever your correct kernel version is, of course. And make sure the version matches in the other commands as well.
Also I got the second command (generating mkinitrd.log) slightly wrong. Please run: sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1
Created attachment 579441 [details] The install.log # rpm -ivh --noscripts kernel-2.6.32-220.7.1.el6.x86_64.rpm kernel-firmware-2.6.32-220.7.1.el6.noarch.rpm Preparing... ########################################### [100%] 1:kernel-firmware ########################################### [ 50%] 2:kernel ########################################### [100%] # sh -x /sbin/new-kernel-pkg --package kernel --install 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1 # sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1
Created attachment 579444 [details] The mkinitrd.log
(In reply to comment #42) > Created attachment 579441 [details] > The install.log > > # rpm -ivh --noscripts kernel-2.6.32-220.7.1.el6.x86_64.rpm > kernel-firmware-2.6.32-220.7.1.el6.noarch.rpm > Preparing... ########################################### [100%] > 1:kernel-firmware ########################################### [ 50%] > 2:kernel ########################################### [100%] > # sh -x /sbin/new-kernel-pkg --package kernel --install > 2.6.32-220.7.1.el6.x86_64 > install.log 2>&1 > # sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod > --update 2.6.32-257.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log > 2>&1 last command has the wrong kernel version!!
Created attachment 579493 [details] The mkinitrd.log Retried with # sh -x /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.32-220.7.1.el6.x86_64 --kernel-args=crashkernel=128M > mkinitrd.log 2>&1
From that log: + doDepmod + '[' -n '' ']' + depmod -ae -F /boot/System.map-2.6.32-220.7.1.el6.x86_64 2.6.32-220.7.1.el6.x86_64 + '[' make == make ']' + mkinitrd + '[' -n --dracut ']' + tool='/sbin/dracut -f /boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 2.6.32-220.7.1.el6.x86_64' As per that, it appears that we're running depmod. If its results aren't there, unless there's some reason to believe otherwise, it seems that this is a depmod bug.
Hi, depmod was leaking memory, the fix in https://bugzilla.redhat.com/show_bug.cgi?id=670613 should make it work again. Do you think it might be a DUP of 670613? Can you test with 6.3 nightlies somehow?
(In reply to comment #48) > Hi, > > depmod was leaking memory, the fix in > https://bugzilla.redhat.com/show_bug.cgi?id=670613 should make it work again. > > Do you think it might be a DUP of 670613? Can you test with 6.3 nightlies > somehow? I don't see RHEL 6.3 available in EC2. If you were able to reproduce the original issue of this bugzilla on some small memory non-EC2, you should be able to check 6.3 as well.
I confirm that where module-init-tools-3.9-17.el6.x86_64 produces # for i in /boot/initramfs*.img ; do echo -n "$i " ; lsinitrd $i | grep lib/modules | wc -l ; done /boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 0 /boot/initramfs-2.6.32-220.el6.x86_64.img 398 after yum upgrade, with module-init-tools-3.9-19.el6_2.x86_64 the bug is fixed: # for i in /boot/initramfs*.img ; do echo -n "$i " ; lsinitrd $i | grep lib/modules | wc -l ; done /boot/initramfs-2.6.32-220.7.1.el6.x86_64.img 398 /boot/initramfs-2.6.32-220.el6.x86_64.img 398 Feel free to close this bug as duplicate of bug 670613, even if you might want to keep this bugzilla open and specifically tested in the EC2 environment.
*** This bug has been marked as a duplicate of bug 670613 ***