Bug 1186359 - F21 Kernel 3.18.3 initramfs hits "error while loading shared libraries: /lib64/libbz2.so.1: cannot read file data: Error 21"
Summary: F21 Kernel 3.18.3 initramfs hits "error while loading shared libraries: /lib6...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 21
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-01-27 14:11 UTC by Bruce Wolfe
Modified: 2019-11-15 08:12 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-03-06 10:12:53 UTC
Type: Bug
Embargoed:
harald: needinfo-


Attachments (Terms of Use)
boot journalctl output (107.72 KB, text/plain)
2015-01-27 14:11 UTC, Bruce Wolfe
no flags Details
debug.log of dracut --debug (16.00 MB, text/plain)
2015-02-15 10:27 UTC, E Stalmans
no flags Details

Description Bruce Wolfe 2015-01-27 14:11:56 UTC
Created attachment 984683 [details]
boot journalctl output

Description of problem:
Latest Kernel update 3.18.3 is not detecting the two LVM partitions that host my /(root) and /home folders - drops to emergency console eventually. 

Booting to the prior kernel 3.17.8 works.


Additional info:
See attached "sos" report

Comment 1 Heinz Mauelshagen 2015-01-27 14:20:22 UTC
Did that happen on a pure update of the kernel packages _or_
after a Fedora update from Fedora 20?

FWIW: the latter happened to me updating from Fedora 20 and I had to 
      reboot a previous, still working kernel and localinstall 3.18.3 to fix it.

Comment 2 Marian Csontos 2015-01-27 14:34:29 UTC
I see few of these:
lvm: error while loading shared libraries: /lib64/libbz2.so.1: cannot read file data: Error 21

I wonder why is libbz2 needed? :-/
prajnoha, is there a missing dependency? Or just missing lib in initramdisk?

Comment 3 Peter Rajnoha 2015-01-27 14:47:42 UTC
(In reply to Marian Csontos from comment #2)
> I see few of these:
> lvm: error while loading shared libraries: /lib64/libbz2.so.1: cannot read
> file data: Error 21

We're not using libbz2, it must be a dependency of the library we're using (libblkid, libudev, libselinux, libreadline, ...).

> 
> I wonder why is libbz2 needed? :-/
> prajnoha, is there a missing dependency? Or just missing lib in initramdisk?

Neither of the two I think - if the library was missing, it would display an error like "cannot be found". This is "cannot read file data" which points to some kind of corruption. I would probably try running fsck and regenerating the initramfs (dracut --force) here.

Comment 4 Peter Rajnoha 2015-01-27 14:50:53 UTC
Seems like the initramfs is incorrectly generated - if it's still possible to boot with the older kernel, do that and then "dracut --force /boot/initramfs-<the new kernel version>.img <the new kernel version>". For proper kernel version string to use, check /usr/lib/modules directory.

Comment 5 Bruce Wolfe 2015-01-27 22:03:01 UTC
(In reply to Heinz Mauelshagen from comment #1)
> Did that happen on a pure update of the kernel packages _or_
> after a Fedora update from Fedora 20?

During a pure update - not a move from Fedora 20 to F21
 
> FWIW: the latter happened to me updating from Fedora 20 and I had to 
>       reboot a previous, still working kernel and localinstall 3.18.3 to fix
> it.

I will try this if an fsck and rebuild of initramfs does not work.

Comment 6 Bruce Wolfe 2015-01-27 22:06:15 UTC
(In reply to Peter Rajnoha from comment #4)
> Seems like the initramfs is incorrectly generated - if it's still possible
> to boot with the older kernel, do that and then "dracut --force
> /boot/initramfs-<the new kernel version>.img <the new kernel version>". For
> proper kernel version string to use, check /usr/lib/modules directory.

I will try this. Although I did try and rebuild the initramfs using mkinitrd (I forgot Fedora is uses dracut), is there a difference between the two tools - i.e. do they make different initramfs files?

Comment 7 Bruce Wolfe 2015-01-28 00:22:18 UTC
I have tried:
1. Rebuild of the initramfs using dracut
2. fsck (except on /(root) - not sure how to when it needs to be unmounted and it is in an LVM volume.) Given that I can boot into my machine using the previous kernel, I am assume that implies the file systems are clean anyway!
3. Reinstalling kernel 3.18.3 (yum remove/install)

Now of these has solved the problem.

SSD disk partitioning:
/dev/sda1 efi
/dev/sda2 /boot
/dev/sda3 LVM2 containing / and /home
/dev/sda4 LVM2 containing /virtStore (virtual machine storage)

Comment 8 Heinz Mauelshagen 2015-01-28 12:16:52 UTC
(In reply to Bruce Wolfe from comment #7)
> I have tried:
> 1. Rebuild of the initramfs using dracut
> 2. fsck (except on /(root) - not sure how to when it needs to be unmounted
> and it is in an LVM volume.) Given that I can boot into my machine using the
> previous kernel, I am assume that implies the file systems are clean anyway!
> 3. Reinstalling kernel 3.18.3 (yum remove/install)
> 
> Now of these has solved the problem.
> 
> SSD disk partitioning:
> /dev/sda1 efi
> /dev/sda2 /boot
> /dev/sda3 LVM2 containing / and /home
> /dev/sda4 LVM2 containing /virtStore (virtual machine storage)

Have you tried a "yum localinstall kernel-3.18.3-201.fc21.x86_64" yet?

To check your root device, boot from a rescue media.

Comment 9 Bruce Wolfe 2015-01-29 00:02:11 UTC
(In reply to Heinz Mauelshagen from comment #8)
> (In reply to Bruce Wolfe from comment #7)
> > I have tried:
> > ...
> 
> Have you tried a "yum localinstall kernel-3.18.3-201.fc21.x86_64" yet?
> 

Yes - no change.

> To check your root device, boot from a rescue media.

Done - as expected, clean.

I also noticed some dev/mapper and lvm updates in the latest set up updates, so applied them without any change either. Still cannot boot into kernel 3.18.3, no change in error reporting.

Comment 10 Bruce Wolfe 2015-01-29 01:43:05 UTC
Following up on a suggestion from ask.fedora.org

Regenerating initramfs again, but this time adding the --no-hostonly as an option and it appears that this solves the problem. I can now successfully boot into kernel 3.18.3

Not sure if you want to continue trying to investigate this as a bug?

I will leave it to someone else here to make that decision.

Comment 11 Harald Hoyer 2015-02-03 10:20:09 UTC
lvm: error while loading shared libraries: /lib64/libbz2.so.1: cannot read file data: Error 21

was clearly the culprit here.

What is the output of:
$ sudo lsinitrd <faulty_image> | grep libbz2


Please also run
$ sudo dracut --debug -f --kver 3.18.3-201.fc21.x86_64 test.img &>debug.log
$ sudo lsinitrd test.img | grep libbz2 &>> debug.log
$ rm test.img

and attach debug.log

Comment 12 Bruce Wolfe 2015-02-14 01:44:51 UTC
I have been away and otherwise busy for the last couple of weeks and my plan was to do the steps asked by Harald Hoyer today - however I logged in to find a new kernel update waiting (3.18.6), applied it and rebooted - successful!

The problem existed for 3.18.3 and 3.18.5, but is fixed in 3.18.6.

Harald, do you want me to take your requested actions anyway? Or shall I close this bug?

Comment 13 E Stalmans 2015-02-15 10:27:38 UTC
Created attachment 991905 [details]
debug.log of dracut --debug

I've had the same problem on 3.18.3-201 and 3.18.6-200. Regenerating initramfs with dracut and --no-hostonly hasn't helped in both cases. 

Attached is the output of dracut --debug and lsinitrd for 3.18.6-200

This was a clean install of F21

Comment 14 Bruce Wolfe 2015-02-20 05:03:27 UTC
Kernel 3.18.7 is also working fine for me - no regression.

Comment 15 E Stalmans 2015-02-20 12:52:43 UTC
(In reply to Bruce Wolfe from comment #14)
> Kernel 3.18.7 is also working fine for me - no regression.

Kernel 3.18.7 working here as well. 

Seems I had an additional issue where XFS and LVM had different ideas about partition size. For some reason this was fine for kernel < 3.17 but 3.18 would break. Resizing partitions and xfs_repair seems to have sorted it out.

Comment 16 Harald Hoyer 2015-03-06 10:12:53 UTC
(In reply to E Stalmans from comment #15)
> (In reply to Bruce Wolfe from comment #14)
> > Kernel 3.18.7 is also working fine for me - no regression.
> 
> Kernel 3.18.7 working here as well. 
> 
> Seems I had an additional issue where XFS and LVM had different ideas about
> partition size. For some reason this was fine for kernel < 3.17 but 3.18
> would break. Resizing partitions and xfs_repair seems to have sorted it out.

well, nothing dracut can fix

Comment 17 Bruce Wolfe 2015-03-15 02:30:31 UTC
This has reappeared in kernel 3.18.9-200.fc21.x86_64

I was able to use dracut --no-hostonly to resolve the issue again.

Should this bug be reopened??

Comment 18 Bruce Wolfe 2015-03-15 02:32:41 UTC
Sorry forgot to flag my last comment.

Comment 19 Oliver Sampson 2015-03-16 08:09:34 UTC
(In reply to Bruce Wolfe from comment #17)
> This has reappeared in kernel 3.18.9-200.fc21.x86_64
> 
> I was able to use dracut --no-hostonly to resolve the issue again.
> 
> Should this bug be reopened??

Well, the last two 3.8.x Fedora 20 updates haven't worked for me, so the real question is: if I update to the latest in the queue (3.18.9-100), and it replaces my 3.17.8 in Grub, will I be able to boot into my machine at all?


Note You need to log in before you can comment on or make changes to this bug.