Bug 491196

Summary: can't boot under kernel-2.6.27.19-170.2.35.fc10.x86_64
Product: [Fedora] Fedora Reporter: Art Werschulz <agw>
Component: mkinitrdAssignee: Peter Jones <pjones>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: low    
Version: 10CC: agw, hdegoede, kernel-maint, musuruan, pjones, rocketraman, wtogami
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-12-18 09:04:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Contents of 2.6.27 initrd missing libreadline and libncurses none

Description Art Werschulz 2009-03-19 19:24:32 UTC
Description of problem:
We cannot boot our 64-bit Intel dual-core machine under the Fedora 10 kernel.

Version-Release number of selected component (if applicable):
kernel-2.6.27.19-170.2.35.fc10.x86_64

How reproducible:
Whenever we try to boot from this kernel, it fails.

Steps to Reproduce:
1. Simply boot from the gvien kernel.
  
Actual results:
When we attempt to boot into kernel-2.6.27.19-170.2.35.fc10.x86_64, we
get the following error msgs:
(1) We get two occurrences of 
       /bin/lvm: error while booting shared libraries: libreadline.so.5:
       cannot open shared object fie: No such file or directory
(2) We next get the msgs
       Unable to access resume device (/dev/VolGroup00/LogVol00)
    and
       mount: could not find filesystem '/dev/root'
This sequence of messages appears twice, after which everything stops.
At this point, a CTL-ALT-DEL will reboot the machine.  We are then able to reboot from an FC9 kernel.

Expected results:
A machine that has successfully booted from the FC10 kernel.

Additional info:
This was part of a network upgrade of our Linux machines.  The rest are all i686 machines, and their upgrade proceeded without a hitch.  However, our one x86_64 machine could not boot from the new kernel.  So we now have the odd situation in which all the software comes from FC10 RPMs, except for the kernel, which is from an FC9 RPM.

Comment 1 Raman Gupta 2009-03-23 05:26:07 UTC
Created attachment 336243 [details]
Contents of 2.6.27 initrd missing libreadline and libncurses

Comment 2 Raman Gupta 2009-03-23 05:27:06 UTC
I can confirm the same error on my 64-bit Intel machine. The previous kernel I had loaded, 2.6.27.15-170.2.24.fc10.x86_64, works just fine.

I checked and compared the contents of the 2.6.25 and 2.6.27 initrd e.g.:

# gzip -cd initrd-2.6.27.15-170.2.24.fc10.x86_64.img | cpio -t -v > initrd-2.6.27.15-170.2.24.fc10.x86_64.img.contents

# gzip -cd initrd-2.6.27.19-170.2.35.fc10.x86_64.img | cpio -t -v > initrd-2.6.27.19-170.2.35.fc10.x86_64.img.contents

I have attached the contents of the 2.6.27.19 initrd on my machine.

It appears the initrd generated for the new kernel is missing a couple of libraries that are present in the old initrd:

lib64/libncurses.so.5.6
lib64/libreadline.so.5.2

Hope that helps!

Comment 3 Art Werschulz 2009-03-23 19:41:12 UTC
I also needed to grab the following libraries from the old version:
lib64/libtinfo.so.5
lib64/libtinfo.so.5.6
That did the trick.

Thanks.

Comment 4 Hans de Goede 2009-03-26 09:24:06 UTC
Hmm,

What does "ldd /sbin/lvm" show as output ?

And what does "ls -l /lib64/libncurses* /lib64/libreadline* /lib64/libtinfo*" 
give as output ?

Comment 5 Raman Gupta 2009-03-26 13:40:33 UTC
# ldd /sbin/lvm
        linux-vdso.so.1 =>  (0x00007fffd55ff000)
        libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000000000110000)
        libreadline.so.5 => /lib64/libreadline.so.5 (0x00007f9ecd2b4000)
        librt.so.1 => /lib64/librt.so.1 (0x0000000000beb000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9ecd097000)
        libsepol.so.1 => /lib64/libsepol.so.1 (0x00007f9ecce5d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f9eccc58000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00007f9ecca36000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f9ecc6c4000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00007f9ecc4a2000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9ecc286000)
        /lib64/ld-linux-x86-64.so.2 (0x00000000006ae000)

# ls -l /lib64/libncurses* /lib64/libreadline* /lib64/libtinfo*
lrwxrwxrwx 1 root root     17 2008-11-26 18:41 /lib64/libncurses.so.5 -> libncurses.so.5.6
-rwxr-xr-x 1 root root 139544 2008-10-02 09:03 /lib64/libncurses.so.5.6
lrwxrwxrwx 1 root root     18 2008-11-26 18:41 /lib64/libncursesw.so.5 -> libncursesw.so.5.6
-rwxr-xr-x 1 root root 189064 2008-10-02 09:03 /lib64/libncursesw.so.5.6
lrwxrwxrwx 1 root root     18 2008-11-26 18:45 /lib64/libreadline.so.5 -> libreadline.so.5.2
-rwxr-xr-x 1 root root 257048 2008-03-23 16:45 /lib64/libreadline.so.5.2
lrwxrwxrwx 1 root root     15 2008-11-26 18:41 /lib64/libtinfo.so.5 -> libtinfo.so.5.6
-rwxr-xr-x 1 root root 131896 2008-10-02 09:03 /lib64/libtinfo.so.5.6

Comment 6 Raman Gupta 2009-03-30 23:19:25 UTC
One more piece of information... when I rebuilt the initrd manually using mkinitrd it worked fine, and included the missing libs.

Comment 7 Raman Gupta 2009-04-11 18:50:33 UTC
I had the same problem after I installed 2.6.27.21-170.2.56.fc10.x86_64, though with different libraries. Once I rebuilt the initrd manually with mkinitrd it was fine.

Contents of initrd built manually (works ok):
http://gist.github.com/93670

Contents of initrd as installed by yum (does not work, could not find libnash):
http://gist.github.com/93672

Here is the diff -u between them:
http://gist.github.com/93673

Comment 8 Jeremy Katz 2009-05-06 16:50:28 UTC
Are you still seeing problems with more recent kernels?

Comment 9 Raman Gupta 2009-05-06 17:06:43 UTC
According to yum, the latest available release kernel on F10 is 2.6.27.21-170.2.56.fc10.x86_64, and as per comment #7, yes I still had the same problem with that kernel.

I'll follow up again as soon as I have a newer kernel available to me via yum.

Comment 10 Chuck Ebbert 2009-05-31 19:55:08 UTC
(In reply to comment #9)
> According to yum, the latest available release kernel on F10 is
> 2.6.27.21-170.2.56.fc10.x86_64, and as per comment #7, yes I still had the same
> problem with that kernel.
> 
> I'll follow up again as soon as I have a newer kernel available to me via yum.  

A 2.6.27.24 kernel was released on May 25...

Do you have the update logs from the previous kernel updates? I'd like to see what other packages were updated along with the kernel.

Comment 11 Raman Gupta 2009-06-01 02:13:23 UTC
(In reply to comment #10)
> A 2.6.27.24 kernel was released on May 25...

Will update this in the next couple of days... do you want me to update just the kernel by itself to avoid any possible issues with other updates? (I've never had a problem in the past combining updates with other packages)

> Do you have the update logs from the previous kernel updates? I'd like to see
> what other packages were updated along with the kernel.  

My yum update log from previous kernel update:

http://gist.github.com/121137

Is there another log that might have more information?

Comment 12 Chuck Ebbert 2009-06-03 07:32:58 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > A 2.6.27.24 kernel was released on May 25...
> 
> Will update this in the next couple of days... do you want me to update just
> the kernel by itself to avoid any possible issues with other updates? (I've
> never had a problem in the past combining updates with other packages)
> 

Yes, try just updating the kernel.

Comment 13 Raman Gupta 2009-06-03 13:39:29 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > > A 2.6.27.24 kernel was released on May 25...
> > 
> > Will update this in the next couple of days... do you want me to update just
> > the kernel by itself to avoid any possible issues with other updates? (I've
> > never had a problem in the past combining updates with other packages)
> > 
> 
> Yes, try just updating the kernel.  

Too late! Just updated to 2.6.27.24-170.2.68.fc10.x86_64 last night in a batch with a bunch of other updates. I rebooted this morning and can report that it worked normally.

Log here:

http://gist.github.com/122985

Comment 14 Raman Gupta 2009-06-18 20:19:52 UTC
Ok, maybe dealing with two separate bugs here but I'm not sure -- while my x86_64 computer boots ok with 2.6.27.24-170.2.68.fc10.x86_64, my other i686 computer cannot boot any kernel after 2.6.27.9-159.

Even rebuilding the mkinitrd on that computer does not fix the issue -- the img is noticeably smaller after 2.6.27.9-159 (3.18m vs 3.67m).

Also, when building the mkinitrd I get the following warning message while building:

resolvedevice: device spec expected

I can't debug too much right now since I am working but will try to report more details later, such as an initrd compare.

Comment 15 Raman Gupta 2009-06-18 22:43:44 UTC
After comparing the initrd from 2.6.27.9-159 with 2.6.27.24-170, I find that the newer kernel is missing the following:

bin/lvm
etc/lvm
etc/lvm/lvm.conf
lib/libreadline.so.5.2
lib/libreadline.so.5 -> /lib/libreadline.so.5.2
lib/libncurses.so.5.6
lib/libncurses.so.5 -> /lib/libncurses.so.5.6
lib/libtinfo.so.5.6
lib/libtinfo.so.5 -> /lib/libtinfo.so.5.6

LVM was active and running fine when I rebuilt the mkinitrd under the 2.6.27.9 kernel.

Here is my fstab:

#
# /etc/fstab
# Created by anaconda on Fri Dec 26 07:39:28 2008
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or vol_id(8) for more info
#
#UUID=4bf5da23-61be-4772-9d80-97cfbc7a809a /                       ext3    defaults        1 1
#UUID=c62c16a9-8e07-4529-942d-a4bbe6bb1596 /boot                   ext3    defaults        1 2
LABEL="/"               /                       ext3    defaults,relatime        1 1
LABEL="/boot"           /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0
LABEL="/home"           /home                   ext3    defaults,relatime        1 2
LABEL=SWAP-sda2         swap                    swap    defaults,pri=1  0 0
LABEL=SWAP-sdb3         swap                    swap    defaults,pri=1  0 0
LABEL=SWAP-sdc2         swap                    swap    defaults,pri=-1 0 0

Comment 16 Raman Gupta 2009-10-06 15:53:34 UTC
On my i686 machine on Fedora 10, kernel 2.6.27.35-170.2.94, I am still having the same problem as previously reported in Comment #14 and #15. Summary: kernel does not boot. Rebuilding initrd using mkinitrd does not help -- initrd size is noticeably smaller and is missing several libraries.

I am forced to run kernel 2.6.27.9-159, which is the last kernel that works on this machine.

Comment 17 Raman Gupta 2009-10-28 17:07:37 UTC
As of the updated kernel 2.6.27.37-170.2.104.fc10.i686, the update worked on my x86_64 machine, but still failed on my i686 machine.

I updated the kernel on its own but that did not solve the problem. Here is the yum log:

Oct 28 12:42:41 Updated: kernel-firmware-2.6.27.37-170.2.104.fc10.noarch
Oct 28 12:42:46 Updated: kernel-headers-2.6.27.37-170.2.104.fc10.i386
Oct 28 12:43:31 Installed: kernel-devel-2.6.27.37-170.2.104.fc10.i686
Oct 28 12:44:20 Installed: kernel-2.6.27.37-170.2.104.fc10.i686
Oct 28 12:44:23 Installed: kmod-nvidia-2.6.27.37-170.2.104.fc10.i686-180.60-1.fc10.6.i686
Oct 28 12:44:23 Updated: kmod-nvidia-180.60-1.fc10.6.i686
Oct 28 12:44:26 Installed: kernel-2.6.27.37-170.2.104.fc10.i686

I then tried removing all non-working kernels and reinstalling. Still did not work. Yum log:

Oct 28 12:46:36 Erased: glibc-headers
Oct 28 12:46:37 Erased: kmod-nvidia-2.6.27.35-170.2.94.fc10.i686
Oct 28 12:46:43 Erased: xorg-x11-drv-nvidia-libs
Oct 28 12:46:44 Erased: kmod-nvidia-2.6.27.37-170.2.104.fc10.i686
Oct 28 12:46:47 Erased: kernel
Oct 28 12:46:50 Erased: kernel-devel
Oct 28 12:46:57 Erased: kernel
Oct 28 12:47:03 Erased: xorg-x11-drv-nvidia
Oct 28 12:47:04 Erased: kernel-headers
Oct 28 12:47:06 Erased: kernel-firmware
Oct 28 12:47:09 Erased: kernel-devel
Oct 28 12:47:14 Erased: kmod-nvidia
Oct 28 12:49:06 Installed: kernel-firmware-2.6.27.37-170.2.104.fc10.noarch
Oct 28 12:52:45 Installed: kernel-devel-2.6.27.37-170.2.104.fc10.i686
Oct 28 12:53:28 Installed: kernel-2.6.27.37-170.2.104.fc10.i686
Oct 28 12:53:29 Installed: kernel-2.6.27.37-170.2.104.fc10.i686

The problem is still the same as in Comment #15 -- the initrd is missing the /bin/lvm, readline, and libcurses libraries.

Comment 18 Raman Gupta 2009-11-13 19:45:39 UTC
As of the updated kernel 2.6.27.38-170.2.113.fc10.i686, the update is still failing to build a working initrd on one of my i686 machines (though I have another one that works fine).

Again, I did the kernel update completely separately from any other updates.

What other information can I provide to resolve this issue?

Comment 19 Bug Zapper 2009-11-18 09:55:36 UTC
This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 20 Bug Zapper 2009-12-18 09:04:42 UTC
Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.