Description of problem: The kernel panics upon boot after printing (paraphrased): nash: can't load libparted-1.8.so.8 Version-Release number of selected component (if applicable): 6.0.71-3.fc10.i386 This only happens for kernels with version > 2.6.27.9-159.fc10.i686 How reproducible: Always Steps to Reproduce: 1.boot a newer kernel 2. 3. Actual results: Kernel panic Expected results: Normal boot. Additional info: This is only happening on an old laptop (IBM thinkpad) It doesn't happen on the 3 other machines I own. I don't know what the difference is.
The latest known kernel which the bug does not appear is 2.6.27.9-117 The exact message is: /bin/nash: error while loading shared libraries: libparted-1.8.so.8: cannot open shared object file: No such file or directory.
Me too. On x86_64 and a i686 boxes, both with software raid. Last working kernel for me is: kernel-2.6.29-0.43.rc2.git1.fc11.x86_64. Tried "yum reinstall parted" and verified that both /lib64 and /usr/lib64 libparted-1.8.so.8 exist. Tried refreshing initrd, no change. Rawhide: mkinitrd-6.0.75-1.fc11.x86_64
What is the output of: ldd -r /sbin/nash On your system?
root@ontap:~# ldd /sbin/nash linux-vdso.so.1 => (0x00007fffa29fe000) libnash.so.6.0.75 => /usr/lib64/libnash.so.6.0.75 (0x0000003ae6000000) libbdevid.so.6.0.75 => /usr/lib64/libbdevid.so.6.0.75 (0x0000003ae6400000) libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000003521c00000) libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000) libblkid.so.1 => /lib64/libblkid.so.1 (0x0000003522400000) libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003520c00000) libsepol.so.1 => /lib64/libsepol.so.1 (0x00000031b5800000) libuuid.so.1 => /lib64/libuuid.so.1 (0x0000003521000000) libpopt.so.0 => /lib64/libpopt.so.0 (0x0000003528e00000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00000031b5000000) libdl.so.2 => /lib64/libdl.so.2 (0x00000031aec00000) libelf.so.1 => /usr/lib64/libelf.so.1 (0x0000003525c00000) libnl.so.1 => /usr/lib64/libnl.so.1 (0x00000031b6c00000) libm.so.6 => /lib64/libm.so.6 (0x00000031ae800000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031b0400000) libc.so.6 => /lib64/libc.so.6 (0x00000031ae400000) libreadline.so.5 => /lib64/libreadline.so.5 (0x00000031caa00000) librt.so.1 => /lib64/librt.so.1 (0x00000031b1800000) libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00000031bf200000) /lib64/ld-linux-x86-64.so.2 (0x00000031ae000000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031af000000) but is that the right nash executable? There are others under /boot root@ontap:boot# find . -name nash ./t/bin/nash ./t1/bin/nash ./t2/bin/nash root@ontap:boot# ldd t*/bin/nash | grep parted libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000) libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000) libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000) but OK, they all use the same libparted. root@ontap:boot# ls -l /usr/lib64/libparted-1.8.so.8 lrwxrwxrwx 1 root root 29 2009-02-04 14:50 /usr/lib64/libparted-1.8.so.8 -> /lib64/libparted-1.8.so.8.0.0
Sorry, you asked for ldd -r... root@ontap:boot# ldd -r /sbin/nash linux-vdso.so.1 => (0x00007fff03dff000) libnash.so.6.0.75 => /usr/lib64/libnash.so.6.0.75 (0x0000003ae6000000) libbdevid.so.6.0.75 => /usr/lib64/libbdevid.so.6.0.75 (0x0000003ae6400000) libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x0000003521c00000) libparted-1.8.so.8 => /usr/lib64/libparted-1.8.so.8 (0x0000003aec600000) libblkid.so.1 => /lib64/libblkid.so.1 (0x0000003522400000) libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003520c00000) libsepol.so.1 => /lib64/libsepol.so.1 (0x00000031b5800000) libuuid.so.1 => /lib64/libuuid.so.1 (0x0000003521000000) libpopt.so.0 => /lib64/libpopt.so.0 (0x0000003528e00000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00000031b5000000) libdl.so.2 => /lib64/libdl.so.2 (0x00000031aec00000) libelf.so.1 => /usr/lib64/libelf.so.1 (0x0000003525c00000) libnl.so.1 => /usr/lib64/libnl.so.1 (0x00000031b6c00000) libm.so.6 => /lib64/libm.so.6 (0x00000031ae800000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031b0400000) libc.so.6 => /lib64/libc.so.6 (0x00000031ae400000) libreadline.so.5 => /lib64/libreadline.so.5 (0x00000031caa00000) librt.so.1 => /lib64/librt.so.1 (0x00000031b1800000) libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00000031bf200000) /lib64/ld-linux-x86-64.so.2 (0x00000031ae000000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031af000000)
Here is mine: ldd -r /sbin/nash linux-gate.so.1 => (0x00110000) libnash.so.6.0.71 => /usr/lib/libnash.so.6.0.71 (0x006f0000) libbdevid.so.6.0.71 => /usr/lib/libbdevid.so.6.0.71 (0x0062a000) libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02 (0x0096a000) libparted-1.8.so.8 => /lib/libparted-1.8.so.8 (0x00984000) libblkid.so.1 => /lib/libblkid.so.1 (0x00ded000) libselinux.so.1 => /lib/libselinux.so.1 (0x0065c000) libsepol.so.1 => /lib/libsepol.so.1 (0x00686000) libuuid.so.1 => /lib/libuuid.so.1 (0x00dfa000) libpopt.so.0 => /lib/libpopt.so.0 (0x0455b000) libresolv.so.2 => /lib/libresolv.so.2 (0x03fe7000) libdl.so.2 => /lib/libdl.so.2 (0x00623000) libelf.so.1 => /usr/lib/libelf.so.1 (0x0430b000) libdhcp-1.99.so.1 => /usr/lib/libdhcp-1.99.so.1 (0x00630000) libnl.so.1 => /usr/lib/libnl.so.1 (0x009f0000) libdhcp4client-4.0.so.0 => /usr/lib/libdhcp4client-4.0.so.0 (0x00a41000) libdhcp6client-1.0.so.2 => /usr/lib/libdhcp6client-1.0.so.2 (0x006c2000) libm.so.6 => /lib/libm.so.6 (0x005f8000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x005bd000) libc.so.6 => /lib/libc.so.6 (0x0070f000) /lib/ld-linux.so.2 (0x005d3000) Note that there used to be a symbolic link in /usr/lib/libparted-1.8.so.8, which I noticed did not belong to any package, and which I deleted in an attempt to fix the problem.
I'm an idiot. I did the ldd command on the wrong machine. My previous comment was an ldd on a machine that boots properly. Here is the one on the broken machine: ldd -r /sbin/nash linux-gate.so.1 => (0x00130000) libnash.so.6.0.71 => /usr/lib/libnash.so.6.0.71 (0x00133000) libbdevid.so.6.0.71 => /usr/lib/libbdevid.so.6.0.71 (0x0014e000) libdevmapper.so.1.02 => /lib/libdevmapper.so.1.02 (0x00152000) libparted-1.8.so.8 => /lib/libparted-1.8.so.8 (0x0016a000) libblkid.so.1 => /lib/libblkid.so.1 (0x001d4000) libselinux.so.1 => /lib/libselinux.so.1 (0x001df000) libsepol.so.1 => /lib/libsepol.so.1 (0x001fb000) libuuid.so.1 => /lib/libuuid.so.1 (0x00235000) libpopt.so.0 => /lib/libpopt.so.0 (0x00239000) libresolv.so.2 => /lib/libresolv.so.2 (0x00242000) libdl.so.2 => /lib/libdl.so.2 (0x00259000) libelf.so.1 => /usr/lib/libelf.so.1 (0x0025e000) libdhcp-1.99.so.1 => /usr/lib/libdhcp-1.99.so.1 (0x00275000) libnl.so.1 => /usr/lib/libnl.so.1 (0x0028a000) libdhcp4client-4.0.so.0 => /usr/lib/libdhcp4client-4.0.so.0 (0x002d9000) libdhcp6client-1.0.so.2 => /usr/lib/libdhcp6client-1.0.so.2 (0x00366000) libm.so.6 => /lib/libm.so.6 (0x00392000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x003bb000) libc.so.6 => /lib/libc.so.6 (0x003c9000) /lib/ld-linux.so.2 (0x00110000)
Weird, that looks just fine. Can you please run (as root): mkinitrd -v test.img $(uname -r) > log and attach the resulting log file here?
Created attachment 331164 [details] log from "mkinitrd -v test.img $(uname -r) > log" for currently working kernel
Created attachment 331165 [details] log2 from "mkinitrd -v /boot/initrd-2.6.29-0.91.rc3.git9.fc11.x86_64.img 2.6.29-0.91.rc3.git9.fc11.x86_64 >log2" latest available kernel, exhibits problem
I presume the test.img from Comment #9 would fail too, but i haven't positively confirmed that. The image from Comment #10 definitely fails. I'd hazard a guess that the problems started with the relocation of libparted in parted-1.8.8-2, but the specific problem isn't jumping out at me in these logs.
Created attachment 331495 [details] console capture Perhaps the problem is that nash is being run before / is mounted? How can nash find shared libs without / ?
(In reply to comment #12) > Created an attachment (id=331495) [details] > console capture > > Perhaps the problem is that nash is being run before / is mounted? > How can nash find shared libs without / ? The libs are (should be) part of the initial ramdisk. Can you please unpack a troublesome initrd like this: mkdir t cd t zcat /boot/initrd.......img | cpio -i And then see (with ls -l) what is under /lib and under /usr/lib ?
Created attachment 331557 [details] ls -lR of broken initrd contents Nope, no libparted in initrd. Also, I notice that lib64/libtinfo.so.5 is a broken link. I don't know if thats significant.
Created attachment 331895 [details] ls -lR of broken initrd libparted is indeed missing from lib and usr/lib
Created attachment 331948 [details] A patch to find /usr/lib libraries, and to actually install them. This hack works for me. Can't help wondering what would be wrong with: ldd -r /sbin/nash | awk '{print $3}' | grep lib instead of about 100 lines of this bash crap!
(In reply to comment #16) > Created an attachment (id=331948) [details] > A patch to find /usr/lib libraries, and to actually install them. > > This hack works for me. > Thanks! Can you explain the second hunk of the patch? Why is that necessary ?
Because it was skipping libparted without it. I'm not sure why it thinks libparted was already installed. I suspect its checking the wrong place? To be honest, I didn't investigate further after it worked.
John, Can you please run "ls -l /lib64/libpart*" and "ls -l /usr/lib64/libpart*" and paste the output here? Thanks!
root@ontap:~# ls -l /lib64/libpart* lrwxrwxrwx 1 root root 29 2009-02-13 21:34 /lib64/libparted-1.8.so.8 -> /lib64/libparted-1.8.so.8.0.0 -rwxr-xr-x 1 root root 388056 2009-01-22 08:37 /lib64/libparted-1.8.so.8.0.0 root@ontap:~# ls -l /usr/lib64/libpart* lrwxrwxrwx 1 root root 12 2009-02-06 14:39 /usr/lib64/libparted-1.8.so.8 -> libparted.so lrwxrwxrwx 1 root root 29 2009-02-04 14:50 /usr/lib64/libparted-1.8.so.8.0.0 -> /lib64/libparted-1.8.so.8.0.0 lrwxrwxrwx 1 root root 34 2009-01-31 14:06 /usr/lib64/libparted.so -> ../../lib64/libparted-1.8.so.8.0.0 root@ontap:~#
I looks like the parted upgrade that moved libparted didn't clean up properly? I did: rm /lib64/libpart* /usr/lib64/libpart* yum reinstall parted now I have: root@ontap:~# ls -l /lib64/libpart* lrwxrwxrwx 1 root root 22 2009-02-16 08:07 /lib64/libparted-1.8.so.8 -> libparted-1.8.so.8.0.0 -rwxr-xr-x 1 root root 385360 2009-01-22 08:37 /lib64/libparted-1.8.so.8.0.0 root@ontap:~# ls -l /usr/lib64/libpart* ls: cannot access /usr/lib64/libpart*: No such file or directory root@ontap:~# Still, mkinitrd shouldn't have been defeated by the /usr/lib64/libparted-1.8.so.8.0.0 softlink.
Even when I create the same softlink on my system I cannot reproduce this, so I'm going to close this as works for me, sorry.
I have the same problem when the initrd is being build libparted is missing! What is the process of getting libparted in the initrd? I have done: yum reinstall parted but after that I still don't get the libparted in my initrd when I reinstall the 2.6.27....-170 kernel. How do I fix this the last kernel update I was able to install was 2.6.27...-134 after that maybe something happened with libparted. How will mkinitrd pick it up?
(In reply to comment #23) > I have the same problem when the initrd is being build libparted is missing! > What is the process of getting libparted in the initrd? > > I have done: yum reinstall parted > > but after that I still don't get the libparted in my initrd when I reinstall > the 2.6.27....-170 kernel. > > How do I fix this the last kernel update I was able to install was > 2.6.27...-134 after that maybe something happened with libparted. How will > mkinitrd pick it up? Hmm, Can you run mkinitrd like this (as root): bash -x /sbin/mkinitrd test.img $(uname -r) &> log And then attach the resulting log file ? Thanks.
Created attachment 345057 [details] mkinitrd script output redirected to log Here is the output of: bash -x /sbin/mkinitrd test.img $(uname -r) &> log I ran this on the machine that will go into a Kernel Panic on boot if a new 2.6.27...170 Kernel update is applied. I ran this with 2.6.27...134 (see log). Hope this helps. Please don't ask to reapply and uninstall the new kernel because that has already once lead to a MBR corruption and so phase 1 of Grub failed so I am very hesitant to apply and roll-back kernels. Anyway hope the log contents will help
Hmm, that log file contains a large bunch of binary data, and it starts with: The default plymouth plugin (.so) doesn't exist Which also is not normal, it looks like your system is thoroughly busted, perhaps it is time to reinstall ? Have you tried running "rpm --verify -a" lately?
Created attachment 345174 [details] Output of rpm --verify -a > rpm_verify.log I don't see anything severely wrong in the verify.log that could have a big impact on the kernel update. What I fail to understand is why the upgrade from kernel 2.6.27....117 to 2.6.27....134 was all fine but when the 2.6.27....159 came along it all came crumbling down. Maybe the new mkinitrd? How does mkinitrd determine what libs it needs and where to find them? I noticed from the yum logs that after my kernel upgrade to 2.6.27...134 last Dec that I got an mkinitrd upgrade to version 6.0.71-3.fc10.i386 on Jan 4th. My guess is that after this it is no longer possible to ugrade Fedora fc10 kernels. Would you recommend rolling back to the default fc10 mkinitrd?
Ok, so I can reproduce this now, and this is a real issue, re-opening and then closing as a dup of the bug were this will be tracked, please see that bug for more details and add yourself to the CC if you want to kept up2date.
*** This bug has been marked as a duplicate of bug 502221 ***