Bug 1131337 - initramfs seems to be corrupted in latest updates
Summary: initramfs seems to be corrupted in latest updates
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 20
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-08-19 04:18 UTC by Ayden Beeson
Modified: 2015-06-29 22:08 UTC (History)
4 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-06-29 22:08:24 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
Broken initramfs - straight from RPM (16.87 MB, application/octet-stream)
2014-08-20 03:02 UTC, Ayden Beeson
no flags Details
lsinitrd - 3.11.10-301.fc20.x86_64 (153.07 KB, text/plain)
2014-08-20 03:03 UTC, Ayden Beeson
no flags Details
lsinitrd - 3.15.10-200.fc20.x86_64 (168.64 KB, text/plain)
2014-08-20 03:04 UTC, Ayden Beeson
no flags Details
Journalctl dracut logs (561.17 KB, text/plain)
2014-08-20 03:23 UTC, Ayden Beeson
no flags Details

Description Ayden Beeson 2014-08-19 04:18:31 UTC
Description of problem:
After updating my fedora 20 install, i receive a kernel panic on boot with a VFS: Unable to mount root FS on unknown block.

Further investigation has shown me an error with the initramfs (initramfs broken padding)

I have attempted to rebuild the initramfs from recovery via dracut (by specifying the kernel version as the recovery kernel is old) and this has now given a initramfs CPIO magic error

Version-Release number of selected component (if applicable):
Fedora 20

How reproducible:
Every time

Steps to Reproduce:
1. Install Fedora 20 KDE (from live USB in my case)
2. Update everything using yum update
3. reboot and let grub select the (default) new kernel

Actual results:
Immediate kernel boot failure after initramfs breaks

Expected results:
Fedora to boot like normal


Additional info:
Falling back to the old kernel works fine, however its very old and is not the way it should need to be.

Comment 1 Harald Hoyer 2014-08-19 08:43:14 UTC
Please attach the output of:
# journalctl --since '-7 days' SYSLOG_IDENTIFIER=dracut
# ls -l /boot/initramfs*.img

And maybe attach the faulty initramfs image.

Also, you should check that boot has sufficient space:

# df -h /boot

To get you back to normal (after attaching the faulty image):

# dracut --regenerate-all --force

or regenerate just the faulty image:

# dracut --force --kver <kernel_version_of_latest_kernel>

Then, check the images:

# ls -lh /boot/initramfs*

and the contents:

# lsinitrd <imagefile>

e.g.

# lsinitrd /boot/initramfs-3.16.0-1.fc22.x86_64.img

Comment 2 Ayden Beeson 2014-08-20 00:34:07 UTC
Hey Harald,

Some comments before i start attaching things:
This is a fresh install, with (close enough to) default partitioning (not LVM or BTRFS though), so /boot has the standard 500mb. Right now with the two kernels and recovery etc it has 26% used.

The last time i regenerated initramfs from the installed OS (even the old version) it corrupted it as well and i had to boot from the live disc, chroot to my install and regenerate it from that to get it back, so i'll avoid regenerating the running kernel (3.11, install kernel) and just mess with the new one (3.15.10-200)

I had this on the previous install of fedora, as well as the one before that on btrfs.

I'll start attaching logs etc now.

Comment 3 Ayden Beeson 2014-08-20 03:02:12 UTC
Created attachment 928589 [details]
Broken initramfs - straight from RPM

This is the broken initramfs that shows up with errors on grub load.

Comment 4 Ayden Beeson 2014-08-20 03:03:33 UTC
Created attachment 928590 [details]
lsinitrd - 3.11.10-301.fc20.x86_64

Comment 5 Ayden Beeson 2014-08-20 03:04:19 UTC
Created attachment 928591 [details]
lsinitrd - 3.15.10-200.fc20.x86_64

Comment 6 Ayden Beeson 2014-08-20 03:07:01 UTC
Comment on attachment 928591 [details]
lsinitrd - 3.15.10-200.fc20.x86_64

lsinitrd for the initramfs that doesnt work.

Comment 7 Ayden Beeson 2014-08-20 03:07:33 UTC
Comment on attachment 928590 [details]
lsinitrd - 3.11.10-301.fc20.x86_64

lsinitrd for the initramfs that does work (base fedora 20 install kernel)

Comment 8 Ayden Beeson 2014-08-20 03:14:54 UTC
I'm running the dracut regeneration now, before i did it:

-rw-------. 1 root root  17M Aug 19 17:00 initramfs-3.15.10-200.fc20.x86_64.img

with in raw bytes was: 17692700

After:
-rw-------. 1 root root  17M Aug 20 13:10 initramfs-3.15.10-200.fc20.x86_64.img
which in raw bytes is: 17699267

rebooting now to check if that has worked, i had previously been running dracut with the file name and kernel version (i.e. : dracut /boot/initramfs-3.15.10-200.fc20.x86_64.img 3.15.10-200.fc20.x86_64) but this time i have used dracut --force --kver 3.15.10-200.fc20.x86_64

Comment 9 Ayden Beeson 2014-08-20 03:19:26 UTC
Ok rebooted, still getting a kernel panic after a failed initram load, got "initramfs broken padding" then a kernel panic with a failure to find a working init.

The only things i have noticed so far is it seems to be the new version of dracut that breaks it, the old version works (but used from the live to redo the initramfs of the stock kernel, plus the originally built version bundled into fedora 20's base.

The last install i did attempt to recreate the base initramfs and it was then broken too, but i'm hoping not to break this one now as i'm using it, so i'll avoid doing that this time.

Comment 10 Ayden Beeson 2014-08-20 03:23:38 UTC
Created attachment 928593 [details]
Journalctl dracut logs

Should show both the original install by me using the previously mentioned command, as well as the new one run using the --kver flag instead.

Comment 11 Harald Hoyer 2014-09-08 12:07:28 UTC
uninstall the dracut-network package
# yum remove dracut-network

and retry.

Comment 12 Ayden Beeson 2014-09-09 23:15:05 UTC
(In reply to Harald Hoyer from comment #11)
> uninstall the dracut-network package
> # yum remove dracut-network
> 
> and retry.

Success! \o/

Any idea why dracut-network is breaking it? I checked /boot before and after the regenerate, with dracut-network installed the initramfs was 17mb, now it is 12mb like the original one....

Either way, that is working now, cheers!

Comment 13 Harald Hoyer 2014-09-12 08:53:12 UTC
Hmm, I think your /boot partition ran out of space.

Comment 14 Ayden Beeson 2014-09-18 04:19:34 UTC
It is not and was definitely not ever full.

As i said above, even with all the given items installed, my boot partition had heaps of free space.

In addition, why would removing dracut-network then fix it? the image was only slightly smaller, so that shouldn't be the case at all.

Here is my df -h for the boot partition at the moment:
df -h /boot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       477M  110M  338M  25% /boot


You'll note the used percentage is only 1% less than it was with the faulty initramfs in there...

Comment 15 Fedora End Of Life 2015-05-29 12:40:23 UTC
This message is a reminder that Fedora 20 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 20. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '20'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 20 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 16 Fedora End Of Life 2015-06-29 22:08:24 UTC
Fedora 20 changed to end-of-life (EOL) status on 2015-06-23. Fedora 20 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.