Bug 482964 - kernel panic: nash can't load libparted
Summary: kernel panic: nash can't load libparted
Keywords:
Status: CLOSED DUPLICATE of bug 502221
Alias: None
Product: Fedora
Classification: Fedora
Component: mkinitrd
Version: 10
Hardware: i386
OS: Linux
low
high
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-01-29 01:52 UTC by Thomas Quinn
Modified: 2009-05-25 08:24 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2009-05-25 08:24:28 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
log from "mkinitrd -v test.img $(uname -r) > log" for currently working kernel (13.39 KB, text/plain)
2009-02-06 19:52 UTC, John Ellson
no flags Details
log2 from "mkinitrd -v /boot/initrd-2.6.29-0.91.rc3.git9.fc11.x86_64.img 2.6.29-0.91.rc3.git9.fc11.x86_64 >log2" latest available kernel, exhibits problem (13.39 KB, text/plain)
2009-02-06 19:54 UTC, John Ellson
no flags Details
console capture (20.21 KB, text/plain)
2009-02-10 23:02 UTC, John Ellson
no flags Details
ls -lR of broken initrd contents (12.77 KB, text/plain)
2009-02-11 12:30 UTC, John Ellson
no flags Details
ls -lR of broken initrd (10.59 KB, text/plain)
2009-02-14 03:34 UTC, Thomas Quinn
no flags Details
A patch to find /usr/lib libraries, and to actually install them. (974 bytes, patch)
2009-02-15 00:22 UTC, John Ellson
no flags Details | Diff
mkinitrd script output redirected to log (265.21 KB, application/octet-stream)
2009-05-22 04:24 UTC, Onno Hommes
no flags Details
Output of rpm --verify -a > rpm_verify.log (20.54 KB, application/octet-stream)
2009-05-23 02:19 UTC, Onno Hommes
no flags Details

Description Thomas Quinn 2009-01-29 01:52:38 UTC
Description of problem:
The kernel panics upon boot after printing (paraphrased):
nash: can't load libparted-1.8.so.8

Version-Release number of selected component (if applicable):
6.0.71-3.fc10.i386
This only happens for kernels with version > 
2.6.27.9-159.fc10.i686

How reproducible:

Always


Steps to Reproduce:
1.boot a newer kernel
2.
3.
  
Actual results:

Kernel panic

Expected results:

Normal boot.

Additional info:
This is only happening on an old laptop (IBM thinkpad) It doesn't happen on the 3 other machines I own.  I don't know what the difference is.

Comment 1 Thomas Quinn 2009-01-29 02:12:49 UTC
The latest known kernel which the bug does not appear is 2.6.27.9-117

The exact message is:
/bin/nash: error while loading shared libraries: libparted-1.8.so.8: cannot open shared object file: No such file or directory.

Comment 2 John Ellson 2009-02-05 02:09:06 UTC
Me too. On x86_64 and a i686 boxes, both with software raid.   Last working kernel for me is: kernel-2.6.29-0.43.rc2.git1.fc11.x86_64.   

Tried "yum reinstall parted" and verified that both /lib64 and /usr/lib64
libparted-1.8.so.8 exist.   Tried refreshing initrd, no change.

Rawhide:  mkinitrd-6.0.75-1.fc11.x86_64

Comment 3 Hans de Goede 2009-02-05 08:23:19 UTC
What is the output of:
ldd -r /sbin/nash

On your system?

Comment 4 John Ellson 2009-02-05 13:06:46 UTC
    root@ontap:~# ldd /sbin/nash
	linux-vdso.so.1 =>  (0x00007fffa29fe000)
	libnash.so.6.0.75 => /usr/lib64/libnash.so.6.0.75 (0x0000003ae6000000)
	libbdevid.so.6.0.75 => /usr/lib64/libbdevid.so.6.0.75 (0x0000003ae6400000)
	libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000003521c00000)
	libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x0000003522400000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003520c00000)
	libsepol.so.1 => /lib64/libsepol.so.1 (0x00000031b5800000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x0000003521000000)
	libpopt.so.0 => /lib64/libpopt.so.0 (0x0000003528e00000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00000031b5000000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031aec00000)
	libelf.so.1 => /usr/lib64/libelf.so.1 (0x0000003525c00000)
	libnl.so.1 => /usr/lib64/libnl.so.1 (0x00000031b6c00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031ae800000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031b0400000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031ae400000)
	libreadline.so.5 => /lib64/libreadline.so.5 (0x00000031caa00000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031b1800000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00000031bf200000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031ae000000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031af000000)


but is that the right nash executable?   There are others under /boot

    root@ontap:boot# find . -name nash
        ./t/bin/nash
        ./t1/bin/nash
        ./t2/bin/nash
    root@ontap:boot# ldd t*/bin/nash | grep parted
	libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000)
	libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000)
	libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000)

but OK, they all use the same libparted.

    root@ontap:boot# ls -l /usr/lib64/libparted-1.8.so.8
        lrwxrwxrwx 1 root root 29 2009-02-04 14:50 /usr/lib64/libparted-1.8.so.8 -> /lib64/libparted-1.8.so.8.0.0

Comment 5 John Ellson 2009-02-05 13:08:17 UTC
Sorry, you asked for ldd -r...

    root@ontap:boot# ldd -r /sbin/nash
	linux-vdso.so.1 =>  (0x00007fff03dff000)
	libnash.so.6.0.75 => /usr/lib64/libnash.so.6.0.75 (0x0000003ae6000000)
	libbdevid.so.6.0.75 => /usr/lib64/libbdevid.so.6.0.75 (0x0000003ae6400000)
	libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000003521c00000)
	libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000)
	libblkid.so.1 => /lib64/libblkid.so.1 (0x0000003522400000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003520c00000)
	libsepol.so.1 => /lib64/libsepol.so.1 (0x00000031b5800000)
	libuuid.so.1 => /lib64/libuuid.so.1 (0x0000003521000000)
	libpopt.so.0 => /lib64/libpopt.so.0 (0x0000003528e00000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00000031b5000000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031aec00000)
	libelf.so.1 => /usr/lib64/libelf.so.1 (0x0000003525c00000)
	libnl.so.1 => /usr/lib64/libnl.so.1 (0x00000031b6c00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031ae800000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031b0400000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031ae400000)
	libreadline.so.5 => /lib64/libreadline.so.5 (0x00000031caa00000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031b1800000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00000031bf200000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031ae000000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031af000000)

Comment 6 Thomas Quinn 2009-02-05 15:50:01 UTC
Here is mine:
ldd -r /sbin/nash
        linux-gate.so.1 =>  (0x00110000)
        libnash.so.6.0.71 => /usr/lib/libnash.so.6.0.71 (0x006f0000)
        libbdevid.so.6.0.71 => /usr/lib/libbdevid.so.6.0.71 (0x0062a000)
        libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02 (0x0096a000)
        libparted-1.8.so.8 => /lib/libparted-1.8.so.8 (0x00984000)
        libblkid.so.1 => /lib/libblkid.so.1 (0x00ded000)
        libselinux.so.1 => /lib/libselinux.so.1 (0x0065c000)
        libsepol.so.1 => /lib/libsepol.so.1 (0x00686000)
        libuuid.so.1 => /lib/libuuid.so.1 (0x00dfa000)
        libpopt.so.0 => /lib/libpopt.so.0 (0x0455b000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x03fe7000)
        libdl.so.2 => /lib/libdl.so.2 (0x00623000)
        libelf.so.1 => /usr/lib/libelf.so.1 (0x0430b000)
        libdhcp-1.99.so.1 => /usr/lib/libdhcp-1.99.so.1 (0x00630000)
        libnl.so.1 => /usr/lib/libnl.so.1 (0x009f0000)
        libdhcp4client-4.0.so.0 => /usr/lib/libdhcp4client-4.0.so.0 (0x00a41000)
        libdhcp6client-1.0.so.2 => /usr/lib/libdhcp6client-1.0.so.2 (0x006c2000)
        libm.so.6 => /lib/libm.so.6 (0x005f8000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x005bd000)
        libc.so.6 => /lib/libc.so.6 (0x0070f000)
        /lib/ld-linux.so.2 (0x005d3000)

Note that there used to be a symbolic link in /usr/lib/libparted-1.8.so.8, which I noticed did not belong to any package, and which I deleted in an attempt to fix the problem.

Comment 7 Thomas Quinn 2009-02-05 15:58:21 UTC
I'm an idiot.  I did the ldd command on the wrong machine.  My previous comment was an ldd on a machine that boots properly.  Here is the one on the broken machine:

ldd -r /sbin/nash
        linux-gate.so.1 =>  (0x00130000)
        libnash.so.6.0.71 => /usr/lib/libnash.so.6.0.71 (0x00133000)
        libbdevid.so.6.0.71 => /usr/lib/libbdevid.so.6.0.71 (0x0014e000)
        libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02 (0x00152000)
        libparted-1.8.so.8 => /lib/libparted-1.8.so.8 (0x0016a000)
        libblkid.so.1 => /lib/libblkid.so.1 (0x001d4000)
        libselinux.so.1 => /lib/libselinux.so.1 (0x001df000)
        libsepol.so.1 => /lib/libsepol.so.1 (0x001fb000)
        libuuid.so.1 => /lib/libuuid.so.1 (0x00235000)
        libpopt.so.0 => /lib/libpopt.so.0 (0x00239000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x00242000)
        libdl.so.2 => /lib/libdl.so.2 (0x00259000)
        libelf.so.1 => /usr/lib/libelf.so.1 (0x0025e000)
        libdhcp-1.99.so.1 => /usr/lib/libdhcp-1.99.so.1 (0x00275000)
        libnl.so.1 => /usr/lib/libnl.so.1 (0x0028a000)
        libdhcp4client-4.0.so.0 => /usr/lib/libdhcp4client-4.0.so.0 (0x002d9000)
        libdhcp6client-1.0.so.2 => /usr/lib/libdhcp6client-1.0.so.2 (0x00366000)
        libm.so.6 => /lib/libm.so.6 (0x00392000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x003bb000)
        libc.so.6 => /lib/libc.so.6 (0x003c9000)
        /lib/ld-linux.so.2 (0x00110000)

Comment 8 Hans de Goede 2009-02-06 15:09:49 UTC
Weird, that looks just fine.

Can you please run (as root):
mkinitrd -v test.img $(uname -r) > log and attach the resulting log file here?

Comment 9 John Ellson 2009-02-06 19:52:03 UTC
Created attachment 331164 [details]
log from "mkinitrd -v test.img $(uname -r) > log" for currently working kernel

Comment 10 John Ellson 2009-02-06 19:54:27 UTC
Created attachment 331165 [details]
log2 from "mkinitrd -v /boot/initrd-2.6.29-0.91.rc3.git9.fc11.x86_64.img 2.6.29-0.91.rc3.git9.fc11.x86_64 >log2"  latest available kernel, exhibits problem

Comment 11 John Ellson 2009-02-06 20:08:37 UTC
I presume the test.img from Comment #9 would fail too, but i haven't positively confirmed that.

The image from Comment #10 definitely fails.

I'd hazard a guess that the problems started with the relocation of libparted in parted-1.8.8-2, but the specific problem isn't jumping out at me in these logs.

Comment 12 John Ellson 2009-02-10 23:02:35 UTC
Created attachment 331495 [details]
console capture

Perhaps the problem is that nash is being run before / is mounted?  
How can nash find shared libs without / ?

Comment 13 Hans de Goede 2009-02-11 08:53:33 UTC
(In reply to comment #12)
> Created an attachment (id=331495) [details]
> console capture
> 
> Perhaps the problem is that nash is being run before / is mounted?  
> How can nash find shared libs without / ?

The libs are (should be) part of the initial ramdisk.

Can you please unpack a troublesome initrd like this:
mkdir t
cd t
zcat /boot/initrd.......img | cpio -i

And then see (with ls -l) what is under /lib and under /usr/lib ?

Comment 14 John Ellson 2009-02-11 12:30:13 UTC
Created attachment 331557 [details]
ls -lR  of broken initrd contents

Nope, no libparted in initrd.

Also, I notice that lib64/libtinfo.so.5 is a broken link.  I don't know if thats significant.

Comment 15 Thomas Quinn 2009-02-14 03:34:28 UTC
Created attachment 331895 [details]
ls -lR of broken initrd

libparted is indeed missing from lib and usr/lib

Comment 16 John Ellson 2009-02-15 00:22:59 UTC
Created attachment 331948 [details]
A patch to find /usr/lib libraries, and to actually install them.

This hack works for me.


Can't help wondering what would be wrong with:
    ldd -r /sbin/nash | awk '{print $3}' | grep lib
instead of about 100 lines of this bash crap!

Comment 17 Hans de Goede 2009-02-15 09:26:48 UTC
(In reply to comment #16)
> Created an attachment (id=331948) [details]
> A patch to find /usr/lib libraries, and to actually install them.
> 
> This hack works for me.
> 

Thanks!

Can you explain the second hunk of the patch? Why is that necessary ?

Comment 18 John Ellson 2009-02-15 12:20:01 UTC
Because it was skipping libparted without it.

I'm not sure why it thinks libparted was already installed.  I suspect its checking the wrong place?  To be honest, I didn't investigate further after
it worked.

Comment 19 Hans de Goede 2009-02-16 12:47:13 UTC
John,

Can you please run "ls -l /lib64/libpart*" and "ls -l /usr/lib64/libpart*" and paste the output here? Thanks!

Comment 20 John Ellson 2009-02-16 13:06:10 UTC
root@ontap:~# ls -l /lib64/libpart*
lrwxrwxrwx 1 root root     29 2009-02-13 21:34 /lib64/libparted-1.8.so.8 -> /lib64/libparted-1.8.so.8.0.0
-rwxr-xr-x 1 root root 388056 2009-01-22 08:37 /lib64/libparted-1.8.so.8.0.0
root@ontap:~# ls -l /usr/lib64/libpart*
lrwxrwxrwx 1 root root 12 2009-02-06 14:39 /usr/lib64/libparted-1.8.so.8 -> libparted.so
lrwxrwxrwx 1 root root 29 2009-02-04 14:50 /usr/lib64/libparted-1.8.so.8.0.0 -> /lib64/libparted-1.8.so.8.0.0
lrwxrwxrwx 1 root root 34 2009-01-31 14:06 /usr/lib64/libparted.so -> ../../lib64/libparted-1.8.so.8.0.0
root@ontap:~#

Comment 21 John Ellson 2009-02-16 13:13:51 UTC
I looks like the parted upgrade that moved libparted didn't clean up properly?

I did:
    rm /lib64/libpart* /usr/lib64/libpart*
    yum reinstall parted

now I have:
root@ontap:~# ls -l /lib64/libpart*
lrwxrwxrwx 1 root root     22 2009-02-16 08:07 /lib64/libparted-1.8.so.8 -> libparted-1.8.so.8.0.0
-rwxr-xr-x 1 root root 385360 2009-01-22 08:37 /lib64/libparted-1.8.so.8.0.0
root@ontap:~# ls -l /usr/lib64/libpart*
ls: cannot access /usr/lib64/libpart*: No such file or directory
root@ontap:~# 


Still, mkinitrd shouldn't have been defeated by the /usr/lib64/libparted-1.8.so.8.0.0 softlink.

Comment 22 Hans de Goede 2009-02-16 14:49:10 UTC
Even when I create the same softlink on my system I cannot reproduce this, so I'm going to close this as works for me, sorry.

Comment 23 Onno Hommes 2009-05-21 02:01:40 UTC
I have the same problem when the initrd is being build libparted is missing! What is the process of getting libparted in the initrd?

I have done: yum reinstall parted

but after that I still don't get the libparted in my initrd when I reinstall the 2.6.27....-170 kernel.

How do I fix this the last kernel update I was able to install was 2.6.27...-134 after that maybe something happened with libparted. How will mkinitrd pick it up?

Comment 24 Hans de Goede 2009-05-21 10:02:55 UTC
(In reply to comment #23)
> I have the same problem when the initrd is being build libparted is missing!
> What is the process of getting libparted in the initrd?
> 
> I have done: yum reinstall parted
> 
> but after that I still don't get the libparted in my initrd when I reinstall
> the 2.6.27....-170 kernel.
> 
> How do I fix this the last kernel update I was able to install was
> 2.6.27...-134 after that maybe something happened with libparted. How will
> mkinitrd pick it up?  

Hmm,

Can you run mkinitrd like this (as root):
bash -x /sbin/mkinitrd test.img $(uname -r) &> log

And then attach the resulting log file ?

Thanks.

Comment 25 Onno Hommes 2009-05-22 04:24:51 UTC
Created attachment 345057 [details]
mkinitrd script output redirected to log

Here is the output of:
bash -x /sbin/mkinitrd test.img $(uname -r) &> log

I ran this on the machine that will go into a Kernel Panic on boot if a new 2.6.27...170 Kernel update is applied. I ran this with 2.6.27...134 (see log).
Hope this helps. Please don't ask to reapply and uninstall the new kernel because that has already once lead to a MBR corruption and so phase 1 of Grub failed so I am very hesitant to apply and roll-back kernels. Anyway hope the log contents will help

Comment 26 Hans de Goede 2009-05-22 06:57:32 UTC
Hmm, that log file contains a large bunch of binary data, and it starts with:
The default plymouth plugin (.so) doesn't exist

Which also is not normal, it looks like your system is thoroughly busted, perhaps it is time to reinstall ?

Have you tried running "rpm --verify -a" lately?

Comment 27 Onno Hommes 2009-05-23 02:19:01 UTC
Created attachment 345174 [details]
Output of rpm --verify -a > rpm_verify.log

I don't see anything severely wrong in the verify.log that could have a big impact on the kernel update. What I fail to understand is why the upgrade from kernel 2.6.27....117 to 2.6.27....134 was all fine but when the 2.6.27....159 came along it all came crumbling down. Maybe the new mkinitrd?

How does mkinitrd determine what libs it needs and where to find them?

I noticed from the yum logs that after my kernel upgrade to 2.6.27...134 last Dec that I got an mkinitrd upgrade to version 6.0.71-3.fc10.i386 on Jan 4th. My guess is that after this it is no longer possible to ugrade Fedora fc10 kernels. Would you recommend rolling back to the default fc10 mkinitrd?

Comment 28 Hans de Goede 2009-05-25 08:23:36 UTC
Ok, so I can reproduce this now, and this is a real issue, re-opening and
then closing as a dup of the bug were this will be tracked, please see that
bug for more details and add yourself to the CC if you want to kept up2date.

Comment 29 Hans de Goede 2009-05-25 08:24:28 UTC

*** This bug has been marked as a duplicate of bug 502221 ***


Note You need to log in before you can comment on or make changes to this bug.