Bug 974000 - fedup: upgrade to F19 can't start without rd.luks.uuid/rd.lvm.lv/rd.md.uuid boot args
fedup: upgrade to F19 can't start without rd.luks.uuid/rd.lvm.lv/rd.md.uuid b...
Status: CLOSED CANTFIX
Product: Fedora
Classification: Fedora
Component: fedup (Show other bugs)
18
i386 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Will Woods
Fedora Extras Quality Assurance
https://fedoraproject.org/wiki/Common...
: CommonBugs
: 970580 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-13 05:09 EDT by Jakub Jedelsky
Modified: 2014-01-22 19:18 EST (History)
20 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-22 19:18:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sos report (78.25 KB, text/plain)
2013-06-13 05:10 EDT, Jakub Jedelsky
no flags Details
List of file in initramfs from an F17->F18 updated machine (196.47 KB, text/plain)
2013-06-27 15:28 EDT, Nicolas Dufresne
no flags Details

  None (edit)
Description Jakub Jedelsky 2013-06-13 05:09:49 EDT
Description of problem:

After I run command for update `fedup --network 19` or `fedup --network 19 --instrepo http://dl.fedoraproject.org/pub/fedora/linux/development/19/i386/os/` and reboot, dracut can't find my encrypted root partition. It hangs on:

"Warning: /dev/mapper/luks-ce7ddc(...) does not exists"

I have two encrypted partitions: / and /home on my laptop. It's without LVM.

sosreport.txt is attached.

Version-Release number of selected component (if applicable):
fedup 0.7.3-4.fc18

How reproducible:
always

Steps to Reproduce:
1. run fedup command
2. reboot and boot into "System update"
3.

Actual results:
Dracut can't find encrypted partition.

Expected results:
Dracut will find encrypted partition and update process will be successful.
Comment 1 Jakub Jedelsky 2013-06-13 05:10:49 EDT
Created attachment 760534 [details]
sos report
Comment 2 Nicolas Dufresne 2013-06-25 18:32:22 EDT
Same here, any known workaround ?
Comment 3 Will Woods 2013-06-26 17:18:34 EDT
Please try fedup-0.7.3-5.fc18 (in the updates repo). Does that fix the problem?
Comment 4 Will Woods 2013-06-26 17:19:20 EDT
Also: is there any prompt to enter the password?
Comment 5 Nicolas Dufresne 2013-06-26 18:04:54 EDT
There was no prompt, though I have fixed that issue by properly porting the rd.luks.uuid= parameter (the all the others) to /etc/default/grub, as mentionned in the wiki. Was this just a workaround ?
Comment 6 Nicolas Dufresne 2013-06-26 18:06:48 EDT
Also, in contrast with Jakob, I do have lvm here. I did not notice that slight difference intially.
Comment 7 Jakub Jedelsky 2013-06-27 03:33:34 EDT
ok, same here. I haven't got a rd.luks.uuid as kernel parameter. After adding this one the upgrade process continue right.
Comment 8 Will Woods 2013-06-27 13:05:06 EDT
I'm glad you got your upgrades working, but I need some more info if I'm actually going to fix the bug!

Could you explain what you mean by "properly porting"? What was missing, or what needed changing? 

Could you attach/paste the boot arguments as set up by fedup, and then show what you had to change to fix it?

Also, what wiki page are you referring to?
Comment 9 Nicolas Dufresne 2013-06-27 13:40:49 EDT
See http://fedoraproject.org/wiki/FedUp section "Grub Updates"

I did not do that step when porting from 17 to 18. But Fedora 18 was figuring out the luks/lvm bits by itself (without any special kernel arguements). But booting into FedUp failed, it did not figure-out there was an encrypted disk without these parameters.

As said, following this extra step worked. The ramfs used in F18 was clearly smarter then the one used in FedUp, though I though they should have been generated using the same tool. I'm fine with having kernel parameters, so I don't fully consider this a bug. Though it's misleading, since it worked for a long time without these parameters.
Comment 10 Will Woods 2013-06-27 14:24:19 EDT
The initramfs for fedup has all the same stuff as your F18 initramfs, with two differences:

  1) it's F19 code instead of F18, and
  2) it's not built on your system.

So either:

  1) something changed so the boot arguments from F18 aren't valid for F19,
     which means we need to migrate those boot arguments, or
  2) there's some file on your system that we need to include in the
     initramfs to make it boot properly.

But I can't tell which is the problem without knowing what the old boot arguments were and/or what extra files (if any) dracut used to put into the initramfs on your F18 system.

So once again: Could you please provide/show your boot arguments from before fedup (e.g. F18), the (incorrect) boot arguments used by fedup, and the fixed boot arguments that allowed the upgrade?

Also, if you have an old (F18) initramfs available, please attach the output of `sudo lsinitrd initramfs-<VERSION>.img`.
Comment 11 Nicolas Dufresne 2013-06-27 15:27:30 EDT
Sorry, I did not cloned my computer before changing the arguments. Though I'm sure they where identical to the default in /etc/default/grub.

Probably something like GRUB_CMDLINE_LINUX="ro quiet rhgb" The rest is generic and you should be able to figure-out.

Option b) is more plausible, but that would mean FedUp should always regenerate it's ramfs in order to copy those local files. In the long term, this seems a source for problems.

The original problem comes from migration from grub1 to grub2, which had to be done manually as described in FedUp wiki page. Not sure how you want to fix that, but, I think you got all the information to understand what happened. I'll attach a output as you asked, but from another machine, which remains F18, and was update from F17, though I did followed the Wiki when I updated this machine, so I should not have this issue if I do update it.

(can't comment and attach in this bugzilla, file list comming)
Comment 12 Nicolas Dufresne 2013-06-27 15:28:45 EDT
Created attachment 766299 [details]
List of file in initramfs from an F17->F18 updated machine

The requested list of files, but from another machine that was ported from F17 to F18 in the past.
Comment 13 Will Woods 2013-06-27 18:01:27 EDT
Actually it seems to be option a). If you have no rd.luks.uuid=<uuid> arguments:

F17 dracut used to automatically try to unlock any crypt devices it found.
F18 dracut won't automatically unlock anything unless you set 'rd.auto=1'..
    but it also adds 'rd.auto=1' to /etc/cmdline.d/01-default.conf in initrd.
F19 dracut (>= 026) drops the 'rd.auto=1', so it won't automatically unlock
    anything.

One possible solution is to have something in fedup examine the system and add the appropriate "rd.luks.uuid=<uuid>" boot arguments.

More generally, fedup needs to provide a way for packages to modify the system and/or boot arguments *before* rebooting to start the upgrade. Then something like dracut could provide the script that generates the boot arguments needed to to get the upgrade to start.
Comment 14 Will Woods 2013-06-28 15:58:11 EDT
(In reply to Nicolas Dufresne from comment #9)
> See http://fedoraproject.org/wiki/FedUp section "Grub Updates"
> 
> I did not do that step when porting from 17 to 18. But Fedora 18 was
> figuring out the luks/lvm bits by itself (without any special kernel
> arguements). But booting into FedUp failed, it did not figure-out there was
> an encrypted disk without these parameters.

Now I get it! You dropped the "rd.luks.uuid=..." bits on the floor (even though the wiki page says they're important), but F18 worked OK without them.

They're required in F19, though, which is why fedup couldn't boot.

> As said, following this extra step worked. The ramfs used in F18 was clearly
> smarter then the one used in FedUp, though I though they should have been
> generated using the same tool.

Nope, they're the same - just that F18 defaults to 'rd.auto=1' and F19 defaults to 'rd.auto=0'. Adding 'rd.auto=1' or the correct 'rd.luks.uuid' arguments would fix the problem.

> I'm fine with having kernel parameters, so I
> don't fully consider this a bug. Though it's misleading, since it worked for
> a long time without these parameters.

And now we know why. Hooray!
Comment 15 Will Woods 2013-06-28 16:19:04 EDT
To clarify: the problem here is that F17 and F18 would boot without specifying 'rd.luks.uuid' or 'rd.lvm.lv' arguments, but F19 won't.

Under normal circumstances, the installer sets up those arguments for you, but some people who migrated from GRUB to GRUB2 by hand have *dropped* those arguments - which works fine for F17 and F18, but fails for F19.

The recommended fix is to restore the rd.luks.uuid/rd.lvm.lv/etc. arguments from your old grub.cfg. In a pinch, you can add "rd.auto=1" to your boot arguments to make dracut try to automatically set up *every* disk it finds; this isn't recommended for general use though.

Whether this is actually a *bug* remains to be decided.
Comment 16 Ralf Ertzinger 2013-07-05 11:16:42 EDT
I've manually dropped the rd.luks.uuid entries from grub because having them interfered with the ability do do discards on them.

This seems to have been fixed for F19, so I'll look into reintroducing them.
Comment 17 Anjo 2013-07-22 08:31:32 EDT
I was upgrading from F18 to F19 and had to add rd.auto=1 to be able get lvm assembled. Without it not pvscan even failed to find anything.
The grub config was automatically set up by a former install so I am not to blame for missing arguments.
Comment 18 Anjo 2013-07-22 08:32:52 EDT
(In reply to Anjo from comment #17)
> I was upgrading from F18 to F19 and had to add rd.auto=1 to be able get lvm
> assembled. Without it not pvscan even failed to find anything.
> The grub config was automatically set up by a former install so I am not to
> blame for missing arguments.

Btw, I am not using LUKS
Comment 19 Michael Cronenworth 2013-08-06 18:37:05 EDT
My F18 kernel line already had rd.luks.uuid/rd.lvm.lv and it was copied over to the F19 Fedup System Update line, but the update process did not prompt me for my password. Adding "rd.auto=1" to the end prompted me for a password and the System Update continued normally.
Comment 20 zimon 2013-08-14 05:52:19 EDT
I've had this problem also in two machines. LVM2 inside LUKS-partition is just not compatible with fedup. Upgrading with fedup always fails. At least the another machine had Fedora 16 initially, was upgraded to F17 and later to F18, but now upgrading to F19 fails. Fedora 16 was installed automatically then without any custom fiddling, so I think this is a rather (frustrating) nasty bug.

Also in F18 using Disks-utility doesn't allow to create a LUKS partition which would have LVM2 and PV+LV partitions inside. Only choice is to make LUKS+ext4 partiton or then unencrypted FAT, NTFS, Ext4, extended or custom type, but custom type doesn't allow to create LUKS+LVM2 either.

So is LUKS+LVM2 really supported or not in F19 and up? Seems it needs frustrating much handwork currently to get to work, when previously it worked out of the box.
Comment 21 Will Woods 2013-08-16 14:40:52 EDT
*** Bug 970580 has been marked as a duplicate of this bug. ***
Comment 22 Urban Widmark 2013-09-07 14:13:56 EDT
I had an issue upgrading F18->F19 where I needed to manually add rd.md.uuid's and rd.lvm.lv for the upgrade to find my devices. I assume that is related to this.

When the default changes from rd.auto=1 to rd.auto=0 fedup will probably not be able to tell if an md device should be added or not (because not all need to be?), although it could warn about any that are not included on the grub commandline. It may also be difficult to tell which lvm volumes that might be missing (check the paths?).

Maybe it could have seen that I used md/lvm at all or for data that is required to boot and issued a warning about the rd.auto change?

Would have save a lot of time instead of having to start with a problem mounting /usr and then work backwards from that.


Note: I did not upgrade my grub, but I did add some devices manually because the F18 installer refused to do anything useful regarding raid+lvm
Comment 23 Kjetil T. Homme 2013-10-22 12:12:04 EDT
I was bitten by the same bug, upgrading from F18 to F19: I was not prompted to enter the passphrase to unlock my harddisk. 

adding auto.rd=1 manually worked beatifully, and after the upgrade the correct value for rd_LUKS_UUID was added to menu.lst automatically (and auto.rd is not mentioned, which I guess means it is disabled).  very nice.

so, based on my experience, adding auto.rd=1 to the kernel parameters for the FedUp boot would be a good idea.  (you can't expect users to search Bugzilla to find workarounds for basic regressions like this.)
Comment 24 zimon 2013-11-07 09:30:39 EST
Fedup seems to be quite unreliable way to upgrade the system.

I finally got it to somewhat upgrade from F18 to F19 using fedup.
Having just auto.rd=1 didn't work for me. I had to give whole works. I have now
in /etc/default/grub

GRUB_CMDLINE_LINUX="rhgb root=/dev/mapper/vg_home-lv_root ro auto.rd=1 rd.lvm.lv=vg_home/lv_root rd.md=0 rd.dm=0 rd.luks.uuid=luks-XYZXYZXY-XYZX-XYZX-XYZX-XYZXYZXYZXYZ

rd.luks.uuid was found from old system-logs.

then:
# grub2-mkconfig -o /etc/grub2.cfg

Before there was GRUB_CMNDLINE_LINUX undefined. It worked fine up and until the kernel-3.9.6-200.fc18.x86_64 in Fedora 18. All later F18 update kernels >= 3.10.x failed to boot also, because they didnt find LVM nor LUKS. And also Fedup trying to upgrade from F18 to F19 failed for the same reason.


OffTopic:

/var/log/fedup.log shows errors. like:
  File "/usr/lib/python2.7/site-packages/fedup/util.py", line 48, in rm_f
    log.warn("failed to remove %s: %s", f, str(e))
NameError: global name 'log' is not defined

 
well, after that system kind of considered itself to be F19, but was between F18 and F19, broken in many places (like kernel, X, Gnome)
 
1) fc19 kernels were not upgraded by fedup, but had to manually install them with yum.
# yum -y update kernel

2) nvidia-drivers weren't upgraded by fedup, so X failed to start. 
# yum -y update kmod-nvidia akmod-nvidia # fixed that.

3) There were 2670+11 packages which fedup failed to upgrade and couldn't be manually updated because of missing dependencies and duplicate packages:

yum: fc18 packages conflict with fc19 packages, like "gamin-0.1.10-14.fc19.x86_64 is a duplicate with gamin-0.1.10-13.fc18.i686"  
NOTE! fc18.i686 and fc19.x86_64

I do not remember why I have both i686 and x86_64 packages installed. Maybe because student version of Matlab which only works in 32bit mode.

# rpm -qa|grep .fc18|wc  # F18 packages
   1792    1792   75568
# rpm -qa|grep .fc19|wc  # F19 packages
   1591    1591   53680


# yum -y update --skip-broken 
...that fixed alot. Have to manually one by one fix the rest.
Comment 25 zimon 2013-11-07 09:36:29 EST
Ah, sorry. Just noticed I had used "auto.rd" and not "rd.auto", so maybe that's was why it didn't work. Well anyway, giving those other rd.-parameters worked. Maybe rd.auto would had found them.
Comment 26 Fedora End Of Life 2013-12-21 08:59:36 EST
This message is a reminder that Fedora 18 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 18. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '18'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 18's end of life.

Thank you for reporting this issue and we are sorry that we may not be 
able to fix it before Fedora 18 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior to Fedora 18's end of life.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 27 Alan Eliasen 2014-01-21 05:36:25 EST
Will Woods,

what is the status of this bug?  It's still marked as "NEW".
Comment 28 Will Woods 2014-01-22 19:18:56 EST
It's complicated, but basically: there's nothing to fix in fedup here.

The actual problem is due to incompatibilities between different versions of dracut; some versions require certain arguments, and other versions don't.

Sure, it's *possible* to make fedup work around that problem, but it doesn't make sense for fedup to be responsible for avoiding other people's bugs. If there's a problem with dracut, we should fix dracut.

And so dracut-033 (in F20) added the '--print-cmdline' flag, which prints the required boot arguments for your system. It'd be pretty easy for dracut/grubby to check the boot arguments and add anything that was missing, but.. that'd be a different bug.



So until that gets fixed, the situation remains the same: this shouldn't happen unless you manually edit your bootloader config and leave something out.

If it does happen, there's a simple workaround: replace the missing options, or add "rd.auto=1" if you don't care about probing every disk on the system.

Since fedup isn't the cause of the problem, and fedup's not the right place to fix it, this bug should be closed.

Note You need to log in before you can comment on or make changes to this bug.