Red Hat Bugzilla – Bug 1013087
initramfs for rescue kernel not generated at system install time, trying to boot rescue kernel causes kernel panic
Last modified: 2013-10-15 12:26:38 EDT
Description of problem: After a successful installation, choosing the rescue kernel option in the GRUB menu results in a kernel panic.
Version-Release number of selected component (if applicable):
anaconda 20.18-1 and anaconda 20.20-1
Steps to Reproduce:
2. Install with default guided partition, LVM scheme.
3. Reboot and choose the rescue kernel
Not a kernel panic.
The initramfs for the rescue kernel is missing.
Could this be a dupe of bug 983287 that slipped through the cracks from F19? Or is it transient?
Proposing as a Fedora 20 beta blocker: "No part of any release-blocking desktop's panel (or equivalent) configuration may crash on startup or be entirely non-functional." And also "A system installed without a graphical package set must boot to a working login prompt without any unintended user intervention, and all virtual consoles intended to provide a working login prompt must do so."
Created attachment 804133 [details]
Created attachment 804134 [details]
Created attachment 804135 [details]
This happens with FC19 and has not yet been fixed.
Is there any news from developers on this?
Discussed this in the 2013-10-02 Blocker Review Meeting . This has been voted a RejectedBlocker. While unfortunate, this does not violate any F20 beta release criteria and thus is rejected as a release blocking issue for f20 beta.
So basically, if I understand correctly, there will be no more rescue kernel and no way to recover a Fedora system.
That is a very serious problem for users of Fedora and fairly frightening as well.
(In reply to KitchM from comment #7)
No. It means this bug will not block the release of beta, should it still be unfixed at the beta go/no-go meeting.
Well then, what about the existing FC19 release's bug?
(In reply to KitchM from comment #9)
Apparently on both F19 and F20, a yum update causes the initramfs to be created for the rescue kernel. I don't know if that's the intended behavior or not. At least on my F19 and F20 system, once I've done a yum update (at least one that includes a kernel update) the rescue kernel has an initramfs and can boot the system.
Not on mine. The kernel panic still occurs.
Is there an initramfs for that kernel? Is there an initrd entry for it in the grub.cfg?
Please read bug at https://bugzilla.redhat.com/show_bug.cgi?id=983287
Anaconda calls (from the program.log):
new-kernel-pkg --mkinitrd --dracut --depmod --update 3.11.0-300.fc20.x86_64
which should create the rescue initrd and set it in the grub.cfg. Reassigning.
So far as I can see this probably isn't grubby either - grubby doesn't do, and isn't supposed to do, anything magical to get rescue kernel properly installed, it is dracut's job (via dracut-config-rescue subpackage). Re-assigning to dracut, as I think that's where the problem must be.
So, a new build of dracut landed in Bodhi quite recently. I thought I'd build a live image including that dracut and see if it fixes the bug. So I did (also included the latest anaconda, 20.24). Install from that image appears to hang the system at 'Installing bootloader' - my VM is showing constant 100% CPU and RAM use and I can't get it to switch to a VT. Interesting, but not entirely helpful...
(In reply to Chris Murphy from comment #2)
> Created attachment 804133 [details]
none of those logs contain the journal or the output of the rpm installation.
Please add the journal of the installation.
is the package "dracut-config-rescue" installed?
(In reply to Harald Hoyer from comment #18)
> is the package "dracut-config-rescue" installed?
yes, it is, otherwise there would be no rescue entry in grub
indeed it is, yeah. I'm currently poking at this in a VM - it's easy enough to reproduce, Harald, just grab any F20 Beta TC2 image ( https://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/ ) and run an install.
(In reply to Adam Williamson from comment #20)
> indeed it is, yeah. I'm currently poking at this in a VM - it's easy enough
> to reproduce, Harald, just grab any F20 Beta TC2 image (
> https://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/ ) and run an install.
Downloaded Fedora-20-Beta-TC2-x86_64-netinst.iso, chose "minimal", autopartitioning.
Reboot, boot into rescue..
So, haraldh and I have tracked this one down, we think. Basically this commit broke it:
Note that it removes a call to 'new-kernel-pkg --rpmposttrans' : that's the problem. That call was added to fix https://bugzilla.redhat.com/show_bug.cgi?id=922988 back in F19 cycle. We're not sure why removing it hasn't had worse consequences (like bringing that bug back), but we think removing it *has* caused this bug. The script that generates rescue initramfs is /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh . This is only run when new-kernel-pkg is called with --rpmposttrans , it's not run with the set of options that anaconda currently calls new-kernel-pkg with.
A kernel update 'fixes' the bug because new-kernel-pkg --rpmposttrans *is* called when a new kernel is installed.
Note that we think this bug is probably restricted to live installs only; in a non-live install, the kernel RPM scripts will be called, so new-kernel-pkg --rpmposttrans will be called that way and the rescue initramfs generated. But in a live image install, as no actual RPM package transactions take place, RPM scripts don't get run.
Bouncing back to anaconda, and to bcl, since he broke it :)
Does that include network installs?
Kitch: no. If you are seeing this with a non-live F19 install it is certainly not due to the above problem. That would be something different, so please track it separately. I haven't yet tested if a normal F19 install has the same symptom for me. The fact that it still doesn't work for you after a 'normal' kernel package install is a further indication that you're seeing a different problem.
And yet all the indications are that it is exactly the same issue. Please read #13 above.
(In reply to Harald Hoyer from comment #22)
It sounds like this bug is restricted to Live Desktop in that case. It definitely always fails, on baremetal or qemu/kvm from Live Desktop.
Sorry if I don't understand; but if you say so.;)
chris: it clearly *is* restricted to live images, and I explained exactly why in c#23. harald and I are both happy with that explanation.
Kitch: um, no, all the indications are that it's completely different.
Your failure affects kernels installed as RPMs after initial system install: the one in this bug does not. In your case, you appear to have an initramfs image for the affected kernel: in this bug, there is not such an image. There's really nothing at all in common between the two bugs except that the highest level symptom is similar: the rescue kernel fails to boot. All the other details are different. Please just follow up in your bug report, and not this one. Thanks.
Created attachment 811343 [details]
Here's what was happening:
I dropped the rpmposttrans because it was making the rescue image show up first. I failed to actually try the rescue boot entry, sorry about that.
The /etc/machine-id from the live image was being copied over, so all the live installs had the same id. I'm now skipping that in the rsync.
livemedia-creator includes the vmlinuz rescue but not the initrd, which was picked up by the bootloader install. that's why the initrd line was missing. livecd-creator shouldn't be including the rescue vmlinuz or initrd. For now I'm also not copying them in the rsync.
rpmposttrans actually needs to be called before the bootloader config is written, it assumes it is the first thing created so it ends up at the start of the boot order. Moved it to be called in postInstall for live. It isn't needed for mirror installs.
I've tested with live Beta TC2 and a netboot.iso install.
anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20 has been submitted as an update for Fedora 20.
Package anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-20.25-1.fc20 python-blivet-0.23-1.fc20 pykickstart-1.99.42-1.fc20'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20 has been pushed to the Fedora 20 stable repository. If problems still persist, please make note of it in this bug report.
Confirm fix in Fedora-20-Beta-TC4-x86_64-DVD.iso which has anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20. I can further vouch for usefulness of this working upon install, in bug 1013767 the regular initramfs causes boot failure with rootfs on thinp, whereas the rescue one succeeds.