Bug 1013087 - initramfs for rescue kernel not generated at system install time, trying to boot rescue kernel causes kernel panic
initramfs for rescue kernel not generated at system install time, trying to b...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
20
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Brian Lane
Fedora Extras Quality Assurance
RejectedBlocker
:
Depends On:
Blocks: F20BetaBlocker
  Show dependency treegraph
 
Reported: 2013-09-27 15:52 EDT by Chris Murphy
Modified: 2013-10-15 12:26 EDT (History)
16 users (show)

See Also:
Fixed In Version: anaconda-20.25-1.fc20
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-15 02:35:22 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
anaconda.log (12.25 KB, text/plain)
2013-09-27 15:58 EDT, Chris Murphy
no flags Details
program.log (57.37 KB, text/plain)
2013-09-27 15:58 EDT, Chris Murphy
no flags Details
storage.log (244.33 KB, text/plain)
2013-09-27 15:58 EDT, Chris Murphy
no flags Details
patch (1.95 KB, patch)
2013-10-11 14:52 EDT, Brian Lane
no flags Details | Diff

  None (edit)
Description Chris Murphy 2013-09-27 15:52:33 EDT
Description of problem: After a successful installation, choosing the rescue kernel option in the GRUB menu results in a kernel panic.


Version-Release number of selected component (if applicable):
Fedora-Live-Desktop-x86_64-20-Alpha-4.iso
anaconda 20.18-1 and anaconda 20.20-1

How reproducible:
Always

Steps to Reproduce:
1. Fedora-Live-Desktop-x86_64-20-Alpha-4.iso
2. Install with default guided partition, LVM scheme.
3. Reboot and choose the rescue kernel

Actual results:
Kernel panic.

Expected results:
Not a kernel panic.

Additional info:
The initramfs for the rescue kernel is missing.
Comment 1 Chris Murphy 2013-09-27 15:58:13 EDT
Could this be a dupe of bug 983287 that slipped through the cracks from F19? Or is it transient?

Proposing as a Fedora 20 beta blocker: "No part of any release-blocking desktop's panel (or equivalent) configuration may crash on startup or be entirely non-functional." And also "A system installed without a graphical package set must boot to a working login prompt without any unintended user intervention, and all virtual consoles intended to provide a working login prompt must do so."
Comment 2 Chris Murphy 2013-09-27 15:58:34 EDT
Created attachment 804133 [details]
anaconda.log
Comment 3 Chris Murphy 2013-09-27 15:58:48 EDT
Created attachment 804134 [details]
program.log
Comment 4 Chris Murphy 2013-09-27 15:58:59 EDT
Created attachment 804135 [details]
storage.log
Comment 5 KitchM 2013-09-30 10:55:10 EDT
This happens with FC19 and has not yet been fixed.

Is there any news from developers on this?
Comment 6 Mike Ruckman 2013-10-02 15:57:03 EDT
Discussed this in the 2013-10-02 Blocker Review Meeting [1]. This has been voted a RejectedBlocker. While unfortunate, this does not violate any F20 beta release criteria and thus is rejected as a release blocking issue for f20 beta.

[1] http://meetbot.fedoraproject.org/fedora-blocker-review/2013-10-02/
Comment 7 KitchM 2013-10-02 16:22:11 EDT
So basically, if I understand correctly, there will be no more rescue kernel and no way to recover a Fedora system.

That is a very serious problem for users of Fedora and fairly frightening as well.
Comment 8 Chris Murphy 2013-10-02 16:34:39 EDT
(In reply to KitchM from comment #7)
No. It means this bug will not block the release of beta, should it still be unfixed at the beta go/no-go meeting.
Comment 9 KitchM 2013-10-02 19:47:06 EDT
Well then, what about the existing FC19 release's bug?
Comment 10 Chris Murphy 2013-10-02 20:12:08 EDT
(In reply to KitchM from comment #9)
Apparently on both F19 and F20, a yum update causes the initramfs to be created for the rescue kernel. I don't know if that's the intended behavior or not. At least on my F19 and F20 system, once I've done a yum update (at least one that includes a kernel update) the rescue kernel has an initramfs and can boot the system.
Comment 11 KitchM 2013-10-04 00:21:23 EDT
Not on mine.  The kernel panic still occurs.
Comment 12 Chris Murphy 2013-10-04 00:31:44 EDT
Is there an initramfs for that kernel? Is there an initrd entry for it in the grub.cfg?
Comment 13 KitchM 2013-10-04 09:37:31 EDT
Please read bug at https://bugzilla.redhat.com/show_bug.cgi?id=983287
Comment 14 Vratislav Podzimek 2013-10-09 09:46:20 EDT
Anaconda calls (from the program.log):

new-kernel-pkg --mkinitrd --dracut --depmod --update 3.11.0-300.fc20.x86_64

which should create the rescue initrd and set it in the grub.cfg. Reassigning.
Comment 15 Adam Williamson 2013-10-11 07:55:27 EDT
So far as I can see this probably isn't grubby either - grubby doesn't do, and isn't supposed to do, anything magical to get rescue kernel properly installed, it is dracut's job (via dracut-config-rescue subpackage). Re-assigning to dracut, as I think that's where the problem must be.
Comment 16 Adam Williamson 2013-10-11 08:37:59 EDT
So, a new build of dracut landed in Bodhi quite recently. I thought I'd build a live image including that dracut and see if it fixes the bug. So I did (also included the latest anaconda, 20.24). Install from that image appears to hang the system at 'Installing bootloader' - my VM is showing constant 100% CPU and RAM use and I can't get it to switch to a VT. Interesting, but not entirely helpful...
Comment 17 Harald Hoyer 2013-10-11 08:39:53 EDT
(In reply to Chris Murphy from comment #2)
> Created attachment 804133 [details]
> anaconda.log

none of those logs contain the journal or the output of the rpm installation.

Please add the journal of the installation.
Comment 18 Harald Hoyer 2013-10-11 08:43:28 EDT
is the package "dracut-config-rescue" installed?
Comment 19 Harald Hoyer 2013-10-11 08:44:20 EDT
(In reply to Harald Hoyer from comment #18)
> is the package "dracut-config-rescue" installed?

yes, it is, otherwise there would be no rescue entry in grub
Comment 20 Adam Williamson 2013-10-11 08:48:20 EDT
indeed it is, yeah. I'm currently poking at this in a VM - it's easy enough to reproduce, Harald, just grab any F20 Beta TC2 image ( https://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/ ) and run an install.
Comment 21 Harald Hoyer 2013-10-11 08:55:39 EDT
(In reply to Adam Williamson from comment #20)
> indeed it is, yeah. I'm currently poking at this in a VM - it's easy enough
> to reproduce, Harald, just grab any F20 Beta TC2 image (
> https://dl.fedoraproject.org/pub/alt/stage/20-Beta-TC2/ ) and run an install.

downloading...
Comment 22 Harald Hoyer 2013-10-11 09:17:50 EDT
Downloaded Fedora-20-Beta-TC2-x86_64-netinst.iso, chose "minimal", autopartitioning.

Reboot, boot into rescue..

WORKSFORME
Comment 23 Adam Williamson 2013-10-11 09:42:13 EDT
So, haraldh and I have tracked this one down, we think. Basically this commit broke it:

https://git.fedorahosted.org/cgit/anaconda.git/commit/pyanaconda/packaging/__init__.py?id=57c19f440148a0f31288bdabd923c4f663ad11fc

Note that it removes a call to 'new-kernel-pkg --rpmposttrans' : that's the problem. That call was added to fix https://bugzilla.redhat.com/show_bug.cgi?id=922988 back in F19 cycle. We're not sure why removing it hasn't had worse consequences (like bringing that bug back), but we think removing it *has* caused this bug. The script that generates rescue initramfs is /etc/kernel/postinst.d/51-dracut-rescue-postinst.sh . This is only run when new-kernel-pkg is called with --rpmposttrans , it's not run with the set of options that anaconda currently calls new-kernel-pkg with.

A kernel update 'fixes' the bug because new-kernel-pkg --rpmposttrans *is* called when a new kernel is installed.

Note that we think this bug is probably restricted to live installs only; in a non-live install, the kernel RPM scripts will be called, so new-kernel-pkg --rpmposttrans will be called that way and the rescue initramfs generated. But in a live image install, as no actual RPM package transactions take place, RPM scripts don't get run.

Bouncing back to anaconda, and to bcl, since he broke it :)
Comment 24 KitchM 2013-10-11 10:01:30 EDT
Does that include network installs?
Comment 25 Adam Williamson 2013-10-11 10:14:10 EDT
Kitch: no. If you are seeing this with a non-live F19 install it is certainly not due to the above problem. That would be something different, so please track it separately. I haven't yet tested if a normal F19 install has the same symptom for me. The fact that it still doesn't work for you after a 'normal' kernel package install is a further indication that you're seeing a different problem.
Comment 26 KitchM 2013-10-11 11:07:14 EDT
And yet all the indications are that it is exactly the same issue.  Please read #13 above.
Comment 27 Chris Murphy 2013-10-11 11:34:37 EDT
(In reply to Harald Hoyer from comment #22)
> WORKSFORME

It sounds like this bug is restricted to Live Desktop in that case. It definitely always fails, on baremetal or qemu/kvm from Live Desktop.
Comment 28 KitchM 2013-10-11 12:24:31 EDT
Sorry if I don't understand; but if you say so.;)
Comment 29 Adam Williamson 2013-10-11 12:53:02 EDT
chris: it clearly *is* restricted to live images, and I explained exactly why in c#23. harald and I are both happy with that explanation.
Comment 30 Adam Williamson 2013-10-11 12:55:47 EDT
Kitch: um, no, all the indications are that it's completely different.

Your failure affects kernels installed as RPMs after initial system install: the one in this bug does not. In your case, you appear to have an initramfs image for the affected kernel: in this bug, there is not such an image. There's really nothing at all in common between the two bugs except that the highest level symptom is similar: the rescue kernel fails to boot. All the other details are different. Please just follow up in your bug report, and not this one. Thanks.
Comment 31 Brian Lane 2013-10-11 14:52:46 EDT
Created attachment 811343 [details]
patch

Here's what was happening:

I dropped the rpmposttrans because it was making the rescue image show up first. I failed to actually try the rescue boot entry, sorry about that.

The /etc/machine-id from the live image was being copied over, so all the live installs had the same id. I'm now skipping that in the rsync.

livemedia-creator includes the vmlinuz rescue but not the initrd, which was picked up by the bootloader install. that's why the initrd line was missing. livecd-creator shouldn't be including the rescue vmlinuz or initrd. For now I'm also not copying them in the rsync.

rpmposttrans actually needs to be called before the bootloader config is written, it assumes it is the first thing created so it ends up at the start of the boot order. Moved it to be called in postInstall for live. It isn't needed for mirror installs.

I've tested with live Beta TC2 and a netboot.iso install.
Comment 32 Fedora Update System 2013-10-11 18:06:29 EDT
anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/FEDORA-2013-18731/python-blivet-0.23-1.fc20,pykickstart-1.99.42-1.fc20,anaconda-20.25-1.fc20
Comment 33 Fedora Update System 2013-10-11 22:27:35 EDT
Package anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-20.25-1.fc20 python-blivet-0.23-1.fc20 pykickstart-1.99.42-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18731/python-blivet-0.23-1.fc20,pykickstart-1.99.42-1.fc20,anaconda-20.25-1.fc20
then log in and leave karma (feedback).
Comment 34 Fedora Update System 2013-10-15 02:35:22 EDT
anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 35 Chris Murphy 2013-10-15 12:26:38 EDT
Confirm fix in Fedora-20-Beta-TC4-x86_64-DVD.iso which has anaconda-20.25-1.fc20, python-blivet-0.23-1.fc20, pykickstart-1.99.42-1.fc20. I can further vouch for usefulness of this working upon install, in bug 1013767 the regular initramfs causes boot failure with rootfs on thinp, whereas the rescue one succeeds.

Note You need to log in before you can comment on or make changes to this bug.