Bug 968540

relevent grub2.cfg



menuentry 'System Upgrade (fedup)' --class fedora --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-53e10dc5-8778-4a55-96ba-b0b7c13553c1' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1'  d2459894-08dd-4fd2-bb40-c0b82842a004
        else
          search --no-floppy --fs-uuid --set=root d2459894-08dd-4fd2-bb40-c0b82842a004
        fi
        echo 'Loading System Upgrade (fedup)'
        linux   /vmlinuz-fedup root=/dev/mapper/luks-af7e726c-5647-447d-8f53-f1bde9f75ca0 ro LANG=en_GB.UTF-8 upgrade systemd.unit=system-upgrade.target enforcing=0
        echo 'Loading initial ramdisk ...'
        initrd /initramfs-fedup.img
}

same bug for me :(

same bug for me :(

Created attachment 759912 [details]
sosreport log

Created attachment 759913 [details]
journalctl

A bit more info about the setup:

I use two partitions, sdb1 holding /boot and sbd2 which is a luks crypt container.
The container contains an physical volume which holds three logical volumes:
/,/home and swap.

When the upgrader boots up, it does not know anything about sdb2 and never asks for a password.
Both fstab and crypttab are empty.

This will (correctly) throw you at the dracut shell when the timeout has reached (30s?).

The root kernel parameter is set to /dev/mapper/luks-94db8421-7547-48c8-939b-db2b16835b7e, which points to dm-5. The uuid is f5fd6fe8-33f4-48f0-b439-742992cbc305.

However, in the dracut shell is the uuid not know.

I can of course open the luks device and then the LVM configuration is know, but I cannot continue from there. Also the uuid does not match the one from F18?

A systemctl default results in doing absolutely nothing.

I believe I am hitting this with F-19 GA after a FedUp from F-18. Happy to attach any debug info and much happier to have a workaround ;)

Well, could we have some details about your disk layout, and the boot parameters you have for your default kernel entry after the upgrade, for a start?

I think the bug is that dracut/systemd generator doesn't look at /etc/cmdline.d, Harald knows the details.

(In reply to Adam Williamson from comment #8)
> Well, could we have some details about your disk layout, and the boot
> parameters you have for your default kernel entry after the upgrade, for a
> start?

I'll attach the sosreport. Does that give you all you need?

Created attachment 767996 [details]
sosreport showing the dracut emergency mode

In case it wasn't clear, the system at no point asks me for LUKS password.

This is very likely the same problem as bug 974000 - you have no 'rd.luks.uuid' boot argument. Fedora 19 dracut won't set up LUKS devices without those arguments.

(Did you upgrade your system from GRUB to GRUB2 by hand, and forget to bring those arguments from your old GRUB config?)

Either way - try adding the old rd.luks.uuid argument to your *existing* F18 kernel entry in grub.cfg. (The installer would have set up for you when you originally installed Fedora). It looks something like: 

   rd.luks.uuid=luks-$UUID_OF_CRYPTO_LUKS_PARTITION

You can find the uuid with 'blkid' if you don't have the old config somewhere.

Then re-run fedup, confirm it has the 'rd.luks.uuid' argument in the fedup boot arguments, and try again.

Does that fix the problem?

*** Bug 980239 has been marked as a duplicate of this bug. ***

(In reply to Will Woods from comment #13)
> This is very likely the same problem as bug 974000 - you have no
> 'rd.luks.uuid' boot argument. Fedora 19 dracut won't set up LUKS devices
> without those arguments.
> 
> (Did you upgrade your system from GRUB to GRUB2 by hand, and forget to bring
> those arguments from your old GRUB config?)
> 
> Either way - try adding the old rd.luks.uuid argument to your *existing* F18
> kernel entry in grub.cfg. (The installer would have set up for you when you
> originally installed Fedora). It looks something like: 
> 
>    rd.luks.uuid=luks-$UUID_OF_CRYPTO_LUKS_PARTITION
> 
> You can find the uuid with 'blkid' if you don't have the old config
> somewhere.
> 
> Then re-run fedup, confirm it has the 'rd.luks.uuid' argument in the fedup
> boot arguments, and try again.
> 
> Does that fix the problem?

Yep, adding rd.luks.uuid did the trick for me. After that, the only issue that I hit post-fedup was that it rebooted into F19 Rescue by default.

I don't recall manually upgrading to grub2 in the past, but I probably fedup'd to F18 pre-release.

I have the same problem.  So why is fedup not creating the proper grub.cfg entry?

1) The installer sets up the correct boot arguments at install time.
2) fedup doesn't modify your boot arguments; it just copies the existing entry, exactly like when you install a new kernel package.

So the boot arguments used by fedup should be the same (correct) boot arguments originally set up by the installer, which should boot just fine, unless:

a) You modified the boot arguments at some point and dropped one or more of the required arguments (frequently while migrating from GRUB to GRUB2 - see bug 974000), or
b) The form of one or more of the required arguments has changed between F18 and F19.

So far I haven't seen the latter case. But if you *do* have 'rd.luks.uuid=XXX' in your boot arguments and fedup doesn't start, please paste your boot commandline here.

Otherwise, I can only conclude that this is basically a duplicate of bug 974000.

Well that is what it did, copied the last Fedora 18 section over which did not have a rd.luks.uuid command.  I do not remember if I ever modified the "linux" line but I certainly wouldn't be surprised if I had as it as it was an fedup upgrade from 17 to 18 and there was a grub to grub2 migration that happened there.  

I had already gotten through the first stage of fedup so I didn't run fudup again but, I was able to get it working by using "blkid | grep crypto" to get the UUID of the / (root) fs.  Then I added the "rd.luks.uuid=luks-<UUID>" to the end of the "linux" line of the "System Upgrade (fedup)" section and rebooted this worked as the upgrade started.  

My only encrypted fs is / (root) so I'm not sure what would happen if others were encrypted as well (we have /opt on another fs).

(In reply to Will Woods from comment #13)

> Then re-run fedup, confirm it has the 'rd.luks.uuid' argument in the fedup
> boot arguments, and try again.

Silly question - how much of fedup needs re-run? The whole thing including network download, etc? Should I remove the menuentry for fedup beforehand?

Hi,

I just ran into the same problem. Fedup wasn't able to find Fedora until I added rd.luks.uuid=luks-$UUID_OF_CRYPTO_LUKS_PARTITION to the fedup boot entry in /boot/grub2/grub.cfg. After adding it the upgrade worked. Rerunning fedup-cli was not necessary.

Note: My whole Fedora is on a encrypted LVM and I upgraded using fedup-cli --iso (because of https://bugzilla.redhat.com/show_bug.cgi?id=877623)

2nd Note: The rd.luks.key switch does not work with the fedup image! Moreover the keyboard layout for entering the key is en_US! I recommend adding a new key (without any special characters, etc.) and removing it after the upgrade is done.

(In reply to Chris Bredesen from comment #19)
> (In reply to Will Woods from comment #13)
> 
> > Then re-run fedup, confirm it has the 'rd.luks.uuid' argument in the fedup
> > boot arguments, and try again.
> 
> Silly question - how much of fedup needs re-run? The whole thing including
> network download, etc? Should I remove the menuentry for fedup beforehand?

You can remove the old entry if you want, but fedup will do it for you.

Just run the same command you ran last time. It will:
  * verify all the previously-downloaded images / packages
  * download only *new* packages (e.g. new updates)
  * remove any stale packages that aren't needed for the upgrade
  * remove the stale bootloader entry
  * add a new, working bootloader entry
and then you should be ready to go.

(In reply to Daniel L. from comment #20)

> I just ran into the same problem. Fedup wasn't able to find Fedora until I
> added rd.luks.uuid=luks-$UUID_OF_CRYPTO_LUKS_PARTITION to the fedup boot
> entry in /boot/grub2/grub.cfg. After adding it the upgrade worked. Rerunning
> fedup-cli was not necessary.

Fedup removes its boot entry before starting the upgrade.

So if you only add rd.luks.uuid=... to the fedup entry, it will be removed before the new 'kernel' package is installed (during the upgrade).

So, depending on how the kernel %post/%posttrans script works, your new system might not have that argument in its boot config. Which means your new system may not be able to boot.

So.. has the upgrade completed? Can you confirm that your new F19 system boots OK?

(In reply to Will Woods from comment #22)
> (In reply to Daniel L. from comment #20)
> 
> > I just ran into the same problem. Fedup wasn't able to find Fedora until I
> > added rd.luks.uuid=luks-$UUID_OF_CRYPTO_LUKS_PARTITION to the fedup boot
> > entry in /boot/grub2/grub.cfg. After adding it the upgrade worked. Rerunning
> > fedup-cli was not necessary.
> 
> Fedup removes its boot entry before starting the upgrade.
> 
> So if you only add rd.luks.uuid=... to the fedup entry, it will be removed
> before the new 'kernel' package is installed (during the upgrade).
> 
> So, depending on how the kernel %post/%posttrans script works, your new
> system might not have that argument in its boot config. Which means your new
> system may not be able to boot.
> 
> So.. has the upgrade completed? Can you confirm that your new F19 system
> boots OK?

The upgrade completed and I was able to boot the system using an old kernel. A new kernel was not installed during the upgrade (maybe because I ran fedup-cli with the --iso option). The boot entries of the old kernels were not changed in /boot/grub2/grub.cfg and worked without the rd.luks.uuid switch.

After rebooting, I ran 'yum update' and updated every package that wasn't updated by fedup. The rd.luks.uuid option was already available in /etc/default/grub and automatically added to the new F19 kernel entry in /boot/grub2/grub.cfg. Now, I wonder why the rd.luks.uuid switch wasn't added to grub.cfg by F18.

For the sake of completeness:

I removed rd.luks.uuid from /etc/default/grub and rebuilt grub.cfg -> F19 still boots with F19 kernel!

In my previous comment I mentioned that F18 didn't add the rd.luks.uuid option from /etc/default/grub to /boot/grub2/grub.cfg -> grub2-mkconfig in F18 does. PackageKit/'yum update' apparently didn't after installing a new kernel.

To close out my little corner of this bug ... I opted for a clean re-install of F-19. My system had been preupgraded and fedup'd a few times (going back AT LEAST) to F-15 and I was on 32-bit, wanted to move to 64. So, I'm removeing myself from the cc but thank you all very much for the collaboration. fedora++

Daniel: "PackageKit/'yum update' apparently didn't after installing a new kernel."

Fedora kernel updates don't re-generate grub2.cfg wrt /etc/default/grub (they don't run grub2-mkconfig): they just copy the most recent entry in grub2.cfg and modify it for the new kernel, same way we did with grub-legacy.

Ultimately it comes down to 'why don't your current kernel entries have the necessary rd.luks.uuid entry in them?', and I'm not sure we have a single answer for that, but it mostly seems to affect 'complex' / long-lived installations. I guess I can add a commonbugs note for this and 974000 saying that if new kernels don't boot after a fedup/yum upgrade to F19, check /etc/default/grub for rd.luks.uuid parameters and add them manually to grub2.cfg, or re-do grub2-mkconfig...

There's an interesting wrinkle in bug 979723:

  https://bugzilla.redhat.com/show_bug.cgi?id=979723#c6

It seems that some lvm.conf settings may have changed between F18 and F19 substantially enough to cause certain systems to fail to boot F19 *after* the upgrade completes successfully.

This is obviously a different problem than *this* bug, but I'm mentioning it here so we can avoid confusing the two issues.

(In reply to Will Woods from comment #17)
> So far I haven't seen the latter case. But if you *do* have
> 'rd.luks.uuid=XXX' in your boot arguments and fedup doesn't start, please
> paste your boot commandline here.

BOOT_IMAGE=/vmlinuz-fedup root=/dev/mapper/fedora_t430-root ro rd.md=0 rd.dm=0 rd.lvm.lv=fedora_t430/root rd.luks.uuid=luks-9c272903-d9d0-4898-bdfe-c53679b43622 rd.lvm.lv=fedora_t430/swap vconsole.keymap=us rhgb quiet LANG=en_US.UTF-8

This boot commandline worked with 18, presenting a graphical password input screen. Then with the fedup initramfs (the above command), it failed to provide any password input.

My drive setup is:
/dev/sda1 is /boot
/dev/sda2 is a luks encrypted lvm2 volume containing root and swap

/dev/sda1: UUID="9c494200-de8f-4ceb-b994-f8c1f6f2548d" TYPE="ext4" 
/dev/sda2: UUID="9c272903-d9d0-4898-bdfe-c53679b43622" TYPE="crypto_LUKS" 

The errors shown were:
"
[   10.364177] localhost kernel: input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input6
[  192.496523] localhost dracut-initqueue[302]: Warning: Could not boot.
[  192.504035] localhost dracut-initqueue[302]: Warning: /dev/fedora_t430/root does not exist
[  192.504232] localhost dracut-initqueue[302]: Warning: /dev/fedora_t430/swap does not exist
[  192.504416] localhost dracut-initqueue[302]: Warning: /dev/mapper/fedora_t430-root does not exist
[  192.507581] localhost systemd[1]: Starting Dracut Emergency Shell...
"

I thought it may have been the order of the options, with the rd.lvm options needing to go after the rd.luks option, but this made no difference. I also tried adding rd.auto=1 to the command as I had seen that this option had changed its default value from 1 to 0 recently. This also made no difference.

I got it to work by removing both rd.lvm boot options. The password input prompt was not graphical like before, but textual. The password was accepted and the system updated to 19.

The system rebooted into 19 and presented the same graphical password input screen that 18 had. Both rd.lvm boot options were present on the 19 initramfs grub commandline.

One difference I noticed was that the 19 initramfs has an /etc/crypttab file filled out with the luks info, while both the 18 and fedup initramfs's did not.

Let me know if you need more info.

I just upgraded from 19 > 20 and the fedup image properly asked for a password in textmode. The upgrade succeeded, so thanks to whoever fixed it, here or elsewhere.

This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Will: is there anything practical to be done here, or shall we let it die?

We can probably close this. This problem should be really unlikely now:

1) Fresh installs of F18 or later will have 'rd.luks...' boot args, and
2) fedup-0.9.x copies /etc/crypttab out of your existing initramfs (if present)

And if it *does* happen, there's not much fedup can do to fix this. It's up to dracut and/or cryptsetup to ensure that the required config data is in the right place(s), if it's not already there.

Closing as CURRENTRELEASE, since the fedup-0.9.x behavior should fix any system that uses /etc/crypttab (or similar config files) instead of boot args.