Bug 822792

Summary: Kernel 3.3.5-2.fc16.i686 fails to mount root at boot time
Product: [Fedora] Fedora Reporter: jd1008
Component: grubbyAssignee: Peter Jones <pjones>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 16CC: bcl, dracut-maint, gansalmon, harald, itamar, jonathan, kernel-maint, madhu.chinakonda, pjones
Target Milestone: ---   
Target Release: ---   
Hardware: i686   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-09-07 23:56:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
boot.log for bug 822792
none
debug output of dracut
none
lzma compressed file from output of command: rpm -qa none

Description jd1008 2012-05-18 07:36:53 UTC 
Description of problem: After running yum update, I rebooted into
the newly installed kernel. It was unable to mount / because
it was not able to find the root device by it's uuid,
and the kernel drops into dracut debug mode.
So, I have to reboot into kernel-3.3.4-3.fc16.i686
which does not have  this problem.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:
Comment 1 Josh Boyer 2012-05-18 13:35:58 UTC 
That sounds more like a dracut problem than a kernel problem.  Can you boot that kernel with rd.info and rd.debug on the kernel command line and provide the output?
Comment 2 jd1008 2012-05-18 19:16:48 UTC 
Created attachment 585470 [details]
boot.log for bug 822792

The attachment is for bug 822792
Comment 3 jd1008 2012-05-18 19:22:48 UTC 
(In reply to comment #2)
> Created attachment 585470 [details]
> boot.log for bug 822792
> 
> The attachment is for bug 822792

Please ignore the attachment.
the boot.log was not for the kernel I tried to boot.
Sorry!! 

So, the problem I face with storing the debug output during boot
is that I am unable to save it, as it scrolls out in real time
and have no way to divert it to a file for later view.
Comment 4 jd1008 2012-05-18 20:47:45 UTC 
Created attachment 585492 [details]
debug output of dracut
Comment 5 jd1008 2012-05-19 15:38:32 UTC 
(In reply to comment #1)
> That sounds more like a dracut problem than a kernel problem.  Can you boot
> that kernel with rd.info and rd.debug on the kernel command line and provide
> the output?

Did attachment 585492 [details] shed any light on why dracut of Kernel 3.3.5-2.fc16.i686
fails to mount root?
Comment 6 Josh Boyer 2012-05-20 12:51:58 UTC 
Can you provide the yum.log that has when you updated the kernel?  That will show what else was updated at the same time and might have gotten pulled into the initramfs that was built for the kernel at install time.
Comment 7 jd1008 2012-05-20 16:19:20 UTC 
(In reply to comment #6)
> Can you provide the yum.log that has when you updated the kernel?  That will
> show what else was updated at the same time and might have gotten pulled
> into the initramfs that was built for the kernel at install time.

The kernel was installed by yum update on 05/12/2012.
Here are the packages updated/installed...

May 12 23:36:45 Updated: sane-backends-libs-1.0.22-10.fc16.i686
May 12 23:36:49 Updated: poppler-0.18.0-3.fc16.i686
May 12 23:36:51 Updated: poppler-devel-0.18.0-3.fc16.i686
May 12 23:36:54 Updated: kernel-headers-3.3.5-2.fc16.i686
May 12 23:37:24 Installed: kernel-devel-3.3.5-2.fc16.i686
May 12 23:38:07 Updated: kernel-doc-3.3.5-2.fc16.noarch
May 12 23:38:11 Updated: tuned-0.2.22-1.fc16.noarch
May 12 23:38:26 Updated: pitivi-0.15.2-1.fc16.noarch
May 12 23:38:27 Updated: 1:yum-updatesd-0.9-12.fc16.noarch
May 12 23:39:04 Updated: system-config-samba-1.2.97-1.fc16.noarch
May 12 23:39:25 Updated: soundconverter-1.5.4-15.fc16.noarch
May 12 23:39:28 Updated: sane-backends-1.0.22-10.fc16.i686
May 12 23:39:29 Updated: sane-backends-drivers-cameras-1.0.22-10.fc16.i686
May 12 23:39:32 Installed: fftw-libs-long-3.3.1-2.fc16.i686
May 12 23:39:34 Installed: fftw-libs-single-3.3.1-2.fc16.i686
May 12 23:39:37 Installed: fftw-libs-quad-3.3.1-2.fc16.i686
May 12 23:39:39 Installed: fftw-libs-double-3.3.1-2.fc16.i686
May 12 23:39:40 Installed: fftw-libs-3.3.1-2.fc16.i686
May 12 23:39:40 Updated: fftw-3.3.1-2.fc16.i686
May 12 23:39:43 Updated: sane-backends-drivers-scanners-1.0.22-10.fc16.i686
May 12 23:39:45 Updated: poppler-glib-0.18.0-3.fc16.i686
May 12 23:39:46 Updated: poppler-qt-0.18.0-3.fc16.i686
May 12 23:40:11 Installed: kernel-3.3.5-2.fc16.i686
May 12 23:40:12 Updated: strace-4.7-1.fc16.i686
May 12 23:40:17 Updated: 1:libguestfs-1.16.21-1.fc16.i686
May 12 23:40:19 Updated: 1:libguestfs-java-1.16.21-1.fc16.i686
May 12 23:40:20 Updated: 1:libguestfs-tools-c-1.16.21-1.fc16.i686
May 12 23:40:21 Updated: 1:ocaml-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:23 Updated: 1:perl-Sys-Guestfs-1.16.21-1.fc16.i686
May 12 23:40:23 Updated: 1:libguestfs-tools-1.16.21-1.fc16.i686
May 12 23:40:24 Updated: 1:ocaml-libguestfs-devel-1.16.21-1.fc16.i686
May 12 23:40:25 Updated: 1:libguestfs-java-devel-1.16.21-1.fc16.i686
May 12 23:40:26 Updated: 1:libguestfs-javadoc-1.16.21-1.fc16.i686
May 12 23:40:27 Updated: 1:ruby-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:28 Updated: 1:libguestfs-devel-1.16.21-1.fc16.i686
May 12 23:40:28 Updated: 1:python-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:29 Updated: poppler-qt-devel-0.18.0-3.fc16.i686
May 12 23:40:30 Updated: sane-backends-devel-1.0.22-10.fc16.i686
May 12 23:40:31 Updated: fftw-devel-3.3.1-2.fc16.i686
May 12 23:40:31 Updated: poppler-utils-0.18.0-3.fc16.i686
May 12 23:40:33 Updated: ibus-hangul-1.4.1-2.fc16.i686
May 12 23:40:39 Updated: monkeystudio-1.9.0.2-1.20120501svn4458.fc16.i686
May 12 23:40:41 Updated: kernel-tools-3.3.5-2.fc16.i686
May 12 23:41:04 Erased: fftw-libs-threads-3.3-3.fc16.i686
May 12 23:41:06 Erased: fftw-libs-openmp-3.3-3.fc16.i686
May 12 23:49:55 Installed: hmaccalc-0.9.12-2.fc15.i686
Comment 8 jd1008 2012-05-20 16:33:43 UTC 
(In reply to comment #6)
> Can you provide the yum.log that has when you updated the kernel?  That will
> show what else was updated at the same time and might have gotten pulled
> into the initramfs that was built for the kernel at install time.

The kernel was installed by yum update on 05/12/2012.
Here are the packages updated/installed...

May 12 23:36:45 Updated: sane-backends-libs-1.0.22-10.fc16.i686
May 12 23:36:49 Updated: poppler-0.18.0-3.fc16.i686
May 12 23:36:51 Updated: poppler-devel-0.18.0-3.fc16.i686
May 12 23:36:54 Updated: kernel-headers-3.3.5-2.fc16.i686
May 12 23:37:24 Installed: kernel-devel-3.3.5-2.fc16.i686
May 12 23:38:07 Updated: kernel-doc-3.3.5-2.fc16.noarch
May 12 23:38:11 Updated: tuned-0.2.22-1.fc16.noarch
May 12 23:38:26 Updated: pitivi-0.15.2-1.fc16.noarch
May 12 23:38:27 Updated: 1:yum-updatesd-0.9-12.fc16.noarch
May 12 23:39:04 Updated: system-config-samba-1.2.97-1.fc16.noarch
May 12 23:39:25 Updated: soundconverter-1.5.4-15.fc16.noarch
May 12 23:39:28 Updated: sane-backends-1.0.22-10.fc16.i686
May 12 23:39:29 Updated: sane-backends-drivers-cameras-1.0.22-10.fc16.i686
May 12 23:39:32 Installed: fftw-libs-long-3.3.1-2.fc16.i686
May 12 23:39:34 Installed: fftw-libs-single-3.3.1-2.fc16.i686
May 12 23:39:37 Installed: fftw-libs-quad-3.3.1-2.fc16.i686
May 12 23:39:39 Installed: fftw-libs-double-3.3.1-2.fc16.i686
May 12 23:39:40 Installed: fftw-libs-3.3.1-2.fc16.i686
May 12 23:39:40 Updated: fftw-3.3.1-2.fc16.i686
May 12 23:39:43 Updated: sane-backends-drivers-scanners-1.0.22-10.fc16.i686
May 12 23:39:45 Updated: poppler-glib-0.18.0-3.fc16.i686
May 12 23:39:46 Updated: poppler-qt-0.18.0-3.fc16.i686
May 12 23:40:11 Installed: kernel-3.3.5-2.fc16.i686
May 12 23:40:12 Updated: strace-4.7-1.fc16.i686
May 12 23:40:17 Updated: 1:libguestfs-1.16.21-1.fc16.i686
May 12 23:40:19 Updated: 1:libguestfs-java-1.16.21-1.fc16.i686
May 12 23:40:20 Updated: 1:libguestfs-tools-c-1.16.21-1.fc16.i686
May 12 23:40:21 Updated: 1:ocaml-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:23 Updated: 1:perl-Sys-Guestfs-1.16.21-1.fc16.i686
May 12 23:40:23 Updated: 1:libguestfs-tools-1.16.21-1.fc16.i686
May 12 23:40:24 Updated: 1:ocaml-libguestfs-devel-1.16.21-1.fc16.i686
May 12 23:40:25 Updated: 1:libguestfs-java-devel-1.16.21-1.fc16.i686
May 12 23:40:26 Updated: 1:libguestfs-javadoc-1.16.21-1.fc16.i686
May 12 23:40:27 Updated: 1:ruby-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:28 Updated: 1:libguestfs-devel-1.16.21-1.fc16.i686
May 12 23:40:28 Updated: 1:python-libguestfs-1.16.21-1.fc16.i686
May 12 23:40:29 Updated: poppler-qt-devel-0.18.0-3.fc16.i686
May 12 23:40:30 Updated: sane-backends-devel-1.0.22-10.fc16.i686
May 12 23:40:31 Updated: fftw-devel-3.3.1-2.fc16.i686
May 12 23:40:31 Updated: poppler-utils-0.18.0-3.fc16.i686
May 12 23:40:33 Updated: ibus-hangul-1.4.1-2.fc16.i686
May 12 23:40:39 Updated: monkeystudio-1.9.0.2-1.20120501svn4458.fc16.i686
May 12 23:40:41 Updated: kernel-tools-3.3.5-2.fc16.i686
May 12 23:41:04 Erased: fftw-libs-threads-3.3-3.fc16.i686
May 12 23:41:06 Erased: fftw-libs-openmp-3.3-3.fc16.i686
May 12 23:49:55 Installed: hmaccalc-0.9.12-2.fc15.i686


In case you do not see the suspicious package in the log,
I will attach the output of rpm -qa   
as all-installed-rpms.txt.xz
Comment 9 jd1008 2012-05-20 16:40:39 UTC 
Created attachment 585666 [details]
lzma compressed file from output of command: rpm -qa
Comment 10 Josh Boyer 2012-05-21 13:45:35 UTC 
I don't see anything in your log that would really impact the initramfs, however it might have been updated in a transaction before the kernel and only included in the new initramfs when the kernel was installed.

Maybe Harald will have a better idea as to what to look for.
Comment 11 jd1008 2012-05-22 05:26:25 UTC 
I just downloaded , installed and booted kernel-3.3.6-3.fc16.i686
and it fails in exactly the same way - it is unable to mount root
device by uuid.
This problem is absent in kernel-3.3.4-3.fc16.i686 and it's
predecessors.

Is this problem unique to fedora?
Comment 12 jd1008 2012-05-27 04:14:20 UTC 
I believe I have fixed  the problem, but I am not 100% certain what had caused it in the first place.
First time it happened was when yum update brought in
 kernel-3.3.5-2.fc16.i686, and I rebooted into the new kernel.
2nd time was when yum update brought in  kernel-3.3.6-3.fc16.i686 .
So I remained with booting kernel-3.3.4-3.fc16.i686.
Finally, today I decided to examine /boot/grub2/grub.cfg.
I read it carefully line by line and compared each line with
the corresponding  line in menuentry for kernel-3.3.4-3.fc16.i686.
I saw that when the first yum update brought in kernel-3.3.5-2.fc16.i686,
it  messed up the uuid's in the menuentry for kernel-3.3.5-2.fc16.i686
as follows:

menuentry 'Fedora (3.3.5-2.fc16.i686)' --class fedora --class gnu-linux --class gnu --class os {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root 03f82b3c-5163-465b-a4c3-3bcb349a1cc4
        echo 'Loading Fedora (3.3.5-2.fc16.i686)'
        linux   /boot/vmlinuz-3.3.5-2.fc16.i686 root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro rd.md=0 rd.lvm=0 rd.dm=0  KEYT
ABLE=us quiet SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8
        echo 'Loading initial ramdisk ...'
        initrd /boot/initramfs-3.3.5-2.fc16.i686.img
}

Notice the difference between the UUID's
The linux line has the correct uuid, but the search line has the wrong uuid.

So, when the 2nd yum update came along, it repeated the same error.

Today, I rpm  -e'd both of the kernels, and did yum update kernel
and it brought in kernel-3.3.6-3.fc16.i686 and I checked grub.cfg.
And now it has the correct uuid on both lines of the the menu entry.
So, I am guessing that installation of 3.3.5-2 must have introduced
the error and 3.3.6-3 simply used that as a template to follow.
I cannot find the binary rpm for 3.3.5-2 to retry installing it to see
if it will commit the same erroneous menuentry into grub.cfg
If you do have it, please send link.
Comment 13 jd1008 2012-06-15 22:39:32 UTC 
The very same problem is now occurring in kernel 
kernel-3.3.8-1.fc16.i686.rpm, except the OS does not drop 
into the dracut shell. It just panics and halts the machine.
Comment 14 jd1008 2012-06-16 04:16:50 UTC 
This is the panic stack trace.
I copied it down on paper because for obvious reasons,
it could not be auto-saved anywhere. So I rebooted into
kernel 3.3.7-1.fc16.i686 and typed the stack into a file.

Panic - not syncing: VFS: Unable to mount root fs on unknown-block (0,0)
Pid: 1, comm: swapper/0 Not tainted 3.3.8-1.fc16.i686.debug #1
Call Trace
c09bcde9     ?  printk+0x2d/0x2f
c09bccb2     panic+0x81/0x18b
c0c87b2a     mount_block_root+0x1b8/0x251
c057162d     ? sys_mknod+0x2d/0x30
c0c87c1c     mount_root+0x59/0x5f
c0c87d41     prepare_namespace+0x11f/0x163
c0560535     ? sys_access+0x25/0x30
c0c878b9     kernel_init+0x146/0x14b
c0c87773     start_kernel+0x385/0x385
c09cf282     kernel_thread_helper+0x86/0x10
Comment 15 jd1008 2012-06-16 04:25:46 UTC 
These are the menuentries for kernel 3.3.8-1.fc16.i686:


menuentry 'Fedora (3.3.8-1.fc16.i686)' --class fedora --class gnu-linux --class gnu --class os {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root 259d7b44-1122-42cc-b504-3dd40027d3b9
        echo 'Loading Fedora (3.3.8-1.fc16.i686)'
        echo 'Loading initial ramdisk ...'
}
menuentry 'Fedora Linux, with Linux 3.3.8-1.fc16.i686 (recovery mode)' --class fedora --class gnu-linux --class gnu --class os {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root 259d7b44-1122-42cc-b504-3dd40027d3b9
        echo 'Loading Fedora (3.3.8-1.fc16.i686)'
        linux   /boot/vmlinuz-3.3.8-1.fc16.i686 root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro single rd.md=0 rd.lvm=0 rd.dm=
0  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
}
menuentry 'Fedora (3.3.8-1.fc16.i686.debug)' --class fedora --class gnu-linux --class gnu --class os {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root 259d7b44-1122-42cc-b504-3dd40027d3b9
        echo 'Loading Fedora (3.3.8-1.fc16.i686.debug)'
        linux   /boot/vmlinuz-3.3.8-1.fc16.i686.debug root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro rd.md=0 rd.lvm=0 rd.dm=0
  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
        echo 'Loading initial ramdisk ...'
}
Comment 16 Josh Boyer 2012-06-16 19:38:51 UTC 
(In reply to comment #15)
> These are the menuentries for kernel 3.3.8-1.fc16.i686:
> 
> 
> menuentry 'Fedora (3.3.8-1.fc16.i686)' --class fedora --class gnu-linux
> --class gnu --class os {
>         load_video
>         set gfxpayload=keep
>         insmod gzio
>         insmod part_msdos
>         insmod ext2
>         set root='(hd0,msdos2)'
>         search --no-floppy --fs-uuid --set=root
> 259d7b44-1122-42cc-b504-3dd40027d3b9
>         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
>         echo 'Loading initial ramdisk ...'
> }

This one is totally broken.  It has no linux or initrd line.

> menuentry 'Fedora Linux, with Linux 3.3.8-1.fc16.i686 (recovery mode)'
> --class fedora --class gnu-linux --class gnu --class os {
>         load_video
>         set gfxpayload=keep
>         insmod gzio
>         insmod part_msdos
>         insmod ext2
>         set root='(hd0,msdos2)'
>         search --no-floppy --fs-uuid --set=root
> 259d7b44-1122-42cc-b504-3dd40027d3b9
>         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
>         linux   /boot/vmlinuz-3.3.8-1.fc16.i686
> root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro single rd.md=0 rd.lvm=0
> rd.dm=
> 0  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0
> LANG=en_US.UTF-8
> }

This one lacks the initrd line.

> menuentry 'Fedora (3.3.8-1.fc16.i686.debug)' --class fedora --class
> gnu-linux --class gnu --class os {
>         load_video
>         set gfxpayload=keep
>         insmod gzio
>         insmod part_msdos
>         insmod ext2
>         set root='(hd0,msdos2)'
>         search --no-floppy --fs-uuid --set=root
> 259d7b44-1122-42cc-b504-3dd40027d3b9
>         echo 'Loading Fedora (3.3.8-1.fc16.i686.debug)'
>         linux   /boot/vmlinuz-3.3.8-1.fc16.i686.debug
> root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro rd.md=0 rd.lvm=0 rd.dm=0
>   KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
>         echo 'Loading initial ramdisk ...'
> }

This one also lacks the initrd line.
Comment 17 jd1008 2012-06-17 18:57:53 UTC 
That is how yum update created the menu entries!!!
Comment 18 jd1008 2012-06-19 22:10:34 UTC 
(In reply to comment #16)
> (In reply to comment #15)
> > These are the menuentries for kernel 3.3.8-1.fc16.i686:
> > 
> > 
> > menuentry 'Fedora (3.3.8-1.fc16.i686)' --class fedora --class gnu-linux
> > --class gnu --class os {
> >         load_video
> >         set gfxpayload=keep
> >         insmod gzio
> >         insmod part_msdos
> >         insmod ext2
> >         set root='(hd0,msdos2)'
> >         search --no-floppy --fs-uuid --set=root
> > 259d7b44-1122-42cc-b504-3dd40027d3b9
> >         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
> >         echo 'Loading initial ramdisk ...'
> > }
> 
> This one is totally broken.  It has no linux or initrd line.
> 
> > menuentry 'Fedora Linux, with Linux 3.3.8-1.fc16.i686 (recovery mode)'
> > --class fedora --class gnu-linux --class gnu --class os {
> >         load_video
> >         set gfxpayload=keep
> >         insmod gzio
> >         insmod part_msdos
> >         insmod ext2
> >         set root='(hd0,msdos2)'
> >         search --no-floppy --fs-uuid --set=root
> > 259d7b44-1122-42cc-b504-3dd40027d3b9
> >         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
> >         linux   /boot/vmlinuz-3.3.8-1.fc16.i686
> > root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro single rd.md=0 rd.lvm=0
> > rd.dm=
> > 0  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0
> > LANG=en_US.UTF-8
> > }
> 
> This one lacks the initrd line.
> 
> > menuentry 'Fedora (3.3.8-1.fc16.i686.debug)' --class fedora --class
> > gnu-linux --class gnu --class os {
> >         load_video
> >         set gfxpayload=keep
> >         insmod gzio
> >         insmod part_msdos
> >         insmod ext2
> >         set root='(hd0,msdos2)'
> >         search --no-floppy --fs-uuid --set=root
> > 259d7b44-1122-42cc-b504-3dd40027d3b9
> >         echo 'Loading Fedora (3.3.8-1.fc16.i686.debug)'
> >         linux   /boot/vmlinuz-3.3.8-1.fc16.i686.debug
> > root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro rd.md=0 rd.lvm=0 rd.dm=0
> >   KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
> >         echo 'Loading initial ramdisk ...'
> > }
> 
> This one also lacks the initrd line.

Hi Josh,
I discovered the problem by accident.
I decided to re-install kernel 3.3.8-1 (yum -y reinstall)
and I observed the error message:
F: Failed to install /usr/lib64/hmaccalc/sha512hmac.hmac

I have no idea what had created /usr/lib64, as I had nothing in it.
So I deleted it, and re-installed the kernel.
All went well, and the menu entry has the right stuff in it
and it boots without any issues.

At least, this issue of kernel panic should be removed from this bug
because it is not related to inability to mount root FS by it's uuid.
Comment 19 Harald Hoyer 2012-07-05 09:35:16 UTC 
(In reply to comment #18)
> (In reply to comment #16)
> > (In reply to comment #15)
> > > These are the menuentries for kernel 3.3.8-1.fc16.i686:
> > > 
> > > 
> > > menuentry 'Fedora (3.3.8-1.fc16.i686)' --class fedora --class gnu-linux
> > > --class gnu --class os {
> > >         load_video
> > >         set gfxpayload=keep
> > >         insmod gzio
> > >         insmod part_msdos
> > >         insmod ext2
> > >         set root='(hd0,msdos2)'
> > >         search --no-floppy --fs-uuid --set=root
> > > 259d7b44-1122-42cc-b504-3dd40027d3b9
> > >         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
> > >         echo 'Loading initial ramdisk ...'
> > > }
> > 
> > This one is totally broken.  It has no linux or initrd line.
> > 
> > > menuentry 'Fedora Linux, with Linux 3.3.8-1.fc16.i686 (recovery mode)'
> > > --class fedora --class gnu-linux --class gnu --class os {
> > >         load_video
> > >         set gfxpayload=keep
> > >         insmod gzio
> > >         insmod part_msdos
> > >         insmod ext2
> > >         set root='(hd0,msdos2)'
> > >         search --no-floppy --fs-uuid --set=root
> > > 259d7b44-1122-42cc-b504-3dd40027d3b9
> > >         echo 'Loading Fedora (3.3.8-1.fc16.i686)'
> > >         linux   /boot/vmlinuz-3.3.8-1.fc16.i686
> > > root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro single rd.md=0 rd.lvm=0
> > > rd.dm=
> > > 0  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0
> > > LANG=en_US.UTF-8
> > > }
> > 
> > This one lacks the initrd line.
> > 
> > > menuentry 'Fedora (3.3.8-1.fc16.i686.debug)' --class fedora --class
> > > gnu-linux --class gnu --class os {
> > >         load_video
> > >         set gfxpayload=keep
> > >         insmod gzio
> > >         insmod part_msdos
> > >         insmod ext2
> > >         set root='(hd0,msdos2)'
> > >         search --no-floppy --fs-uuid --set=root
> > > 259d7b44-1122-42cc-b504-3dd40027d3b9
> > >         echo 'Loading Fedora (3.3.8-1.fc16.i686.debug)'
> > >         linux   /boot/vmlinuz-3.3.8-1.fc16.i686.debug
> > > root=UUID=259d7b44-1122-42cc-b504-3dd40027d3b9 ro rd.md=0 rd.lvm=0 rd.dm=0
> > >   KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
> > >         echo 'Loading initial ramdisk ...'
> > > }
> > 
> > This one also lacks the initrd line.
> 
> Hi Josh,
> I discovered the problem by accident.
> I decided to re-install kernel 3.3.8-1 (yum -y reinstall)
> and I observed the error message:
> F: Failed to install /usr/lib64/hmaccalc/sha512hmac.hmac
> 
> I have no idea what had created /usr/lib64, as I had nothing in it.
> So I deleted it, and re-installed the kernel.
> All went well, and the menu entry has the right stuff in it
> and it boots without any issues.
> 
> At least, this issue of kernel panic should be removed from this bug
> because it is not related to inability to mount root FS by it's uuid.

/usr/lib64/hmaccalc/sha512hmac.hmac

you have dracut-fips installed, are you sure, you want fips enabled on your machine?

if not, please uninstall dracut-fips
Comment 20 jd1008 2012-09-07 23:56:44 UTC 
Close. User error by installing dracut-fips.
/usr/lib64 was deleted and dracut.fips uninstalled
and the probelm has not recurred since