Bug 1012899 - Upgrading to F20 with luks doesn't boot before upgrade
Upgrading to F20 with luks doesn't boot before upgrade
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
Depends On:
  Show dependency treegraph
Reported: 2013-09-27 06:51 EDT by Michael Scherer
Modified: 2015-05-04 11:20 EDT (History)
20 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2015-05-04 11:20:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
sos report of ramdisk (82.86 KB, text/plain)
2013-09-27 06:51 EDT, Michael Scherer
no flags Details
add support for older option for crypsetup generator (1.19 KB, patch)
2014-01-18 04:34 EST, Michael Scherer
no flags Details | Diff
add support for older option for crypsetup generator (1.19 KB, patch)
2014-01-18 04:40 EST, Michael Scherer
no flags Details | Diff

  None (edit)
Description Michael Scherer 2013-09-27 06:51:32 EDT
Created attachment 803891 [details]
sos report of ramdisk

Description of problem:

On a fresh updated F19, with encrypted harddrive ( 2 partitions, 1 for /boot, 1 for encrypted system + lvm ), the initrd generated by fedup doesn't boot.

After a quick investigation, it seems to be because it doesn't detect the luks encrypted volume. Looking at the initrd generated, it doesn't have anything in /etc/cmdline.d/90crypt.conf .

I attach to this bug the rdsosreport.txt
Comment 1 Will Woods 2013-09-27 14:34:30 EDT
What's in /etc/cmdline.d/90crypt.conf? What created that file?
Comment 2 Michael Scherer 2013-09-27 15:56:14 EDT
According to the attached report, this is empty.

I do not know what create the file, maybe that's a dracut problem, so maybe this should be reassigned ?

( but I was unable to pinpoint when and how the initrd is created to further investigate)

And as it seems I forgot to explain what i did, I did run this as root :
# fedup --network 20

and then rebooted on the initrd/kernel. 

Hope that answer the question
Comment 3 Michael Scherer 2013-10-19 19:28:52 EDT
Ok, I found the problem and I have a patch :


Without /etc/crypttab, systemd do not generate the unit, and so the whole initramdisk fail to mount / and so the system cannot continue upgrading.
Comment 4 raphael.brandis 2013-12-17 19:52:51 EST
I can confirm that apparently, this problem is still present. When I try to reboot after completing fedup to perform the actual upgrade from F19 to F20, after entering my LUKS passphrase I land in an emergency console.
There I found some mentions in the logs of /etc/crypttab missing, so I assume it's the same problem, but don't know how to investigate it further. I'll be happy to provide more information if needed.
Comment 5 hostmaster 2013-12-18 06:12:31 EST
It seems dracut never creates a new initrd-fedup.img so Luks cannot work at fedup phase.

/etc/crypttab exists

/etc/sysconfig/grub contains

GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 rd.luks.uuid=luks-*** $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.lvm.lv=vg_sys/usr rd.lvm.lv=vg_sys/root vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_sys/swap vconsole.keymap=de"

Still no success.

fedup needs much more tests...
Comment 6 raphael.brandis 2013-12-18 06:30:19 EST
For me, the upgrade worked when I updated to fedup 0.8 from the upgrades-testing repo.
Comment 7 hostmaster 2013-12-18 14:52:59 EST
Version: fedup-0.8.0-3.fc19.noarch

It does not copy /etc/crypttab into initrd-fedup.img which obviously causes that trouble.
Comment 8 Michal Ambroz 2013-12-20 17:00:34 EST
Same issue here.
Encrypted LVM device.
After reboot I end-up in the emergency console.

I guess the initramfs is not touched at all before reboot. I guess that without cryptsetup it wont work.
# ls -l /boot/initramfs-fedup.img
-rw-r--r--. 1 root root 32346504 2013-12-12 15:05 /boot/initramfs-fedup.img

# md5sum /boot/initramfs-fedup.img 
aa4e10450b0112623b8e9bad8fc3736e  /boot/initramfs-fedup.img
Comment 9 rich 2013-12-21 19:35:46 EST
I am seeing the same thing in upgrading from F19->F20.  I am seeing the same symptoms as described in the last few comments.  I also noted that none of the files in /etc/cmdline.d have any content, they are all 0 length.

Would rebuilding the initramfs-fedup.img help as an interim fix?
Comment 10 Will Woods 2014-01-17 17:40:36 EST
The fedup initramfs isn't generated on your system. It's a generic initramfs that should work for any system, provided you have the necessary boot arguments.

The installer sets up these boot arguments for you, so generally /etc/crypttab isn't needed in this case - unless:

1) you set up LUKS by hand and didn't add those boot arguments, or
2) you (re)wrote grub.cfg/grub2.cfg by hand and left them out


1) Do you have a 'rd.luks.uuid=luks-<UUID>' argument in your grub.cfg/grub2.cfg for each LUKS device?
2) Are those devices listed in /etc/crypttab?
3) Does the upgrade start if you add 'rd.luks.uuid=luks-<UUID>' for everything in /etc/crypttab?

Basically, what's in your /etc/crypttab that the system needs to start up?
Comment 11 Michael Scherer 2014-01-18 04:06:13 EST
I didn't add luks by hand, and I didn't removed grub config to remove them. However, i carried update since I F16 or F15, so it could very well be a left over from that time.

So to answer, question 1 :
Here is what I have in grub.cfg :

title Fedora (3.12.7-300.fc20.x86_64) 20 (Heisenbug)
	root (hd0,0)
	kernel /vmlinuz-3.12.7-300.fc20.x86_64 ro root=/dev/mapper/vg_dhcp19397-lv_root rd_LUKS_UUID=luks-a887dba5-82bb-483f-9b71-e086b41c4c2f rd_LVM_LV=vg_dhcp19397/lv_root rd_LVM_LV=vg_dhcp19397/lv_swap rd_NO_MD rd_NO_DM LANG=fr_FR.utf8 SYSFONT=latarcyrheb-sun16 KEYTABLE=fr rhgb
	initrd /initramfs-3.12.7-300.fc20.x86_64.img

I do not know if rd_LUKS_UUID is 
2) I do have 1 line in /etc/crypttab and this match the UUID :
luks-a887dba5-82bb-483f-9b71-e086b41c4c2f UUID=a887dba5-82bb-483f-9b71-e086b41c4c2f none 

3) I cannot test, sorry, I did the upgrade already using my own patch, and it worked.
Comment 12 Michael Scherer 2014-01-18 04:16:51 EST
Looking a bit at my sosreport, I suspect the issue may be on systemd-cryptsetup-generator side. IE, it only understand /etc/crypttab and rd.luks.uuid, while dracut may understand more ( ie rd_LUKS_UUID ).

Since there is :
[    6.840531] localhost dracut-initqueue[290]: Failed to issue method call: Unit systemd-cryptsetup@luks\x2da887dba5\x2d82bb\x2d483f\x2d9b71\x2de086b41c4c2f.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-cryptsetup@luks\x2da887dba5\x2d82bb\x2d483f\x2d9b71\x2de086b41c4c2f.service' for details.

I think it was just not generated by the generator. 
We can have 3 fixes:
- declare that as unsupported and deprecated 
- modify systemd to make it support the old option
- modify fedup to make it fix the bootload and/or include /etc/crypttab to suit systemd

Option 2 is trivial, I will provide a untested patch, but I would be in favor of not carrying compat forever if possible, and this would only happen if we do fix configuration of people or at least warn them ( and if possible in a way they cannot miss if we cannot fix automatically, cause while I read the release notes, I likely skip a potential warning about "this is deprecated, do not use it" a few releases ago )
Comment 13 Michael Scherer 2014-01-18 04:34:39 EST
Created attachment 851941 [details]
add support for older option for crypsetup generator
Comment 14 Michael Scherer 2014-01-18 04:40:37 EST
Created attachment 851942 [details]
add support for older option for crypsetup generator

slightly fixed the commit message, to list what bugzilla I am talking about
Comment 15 Will Woods 2014-01-21 12:56:36 EST
Ah, so the problem is that systemd-cryptsetup-generator doesn't handle the old rd_LUKS_UUID argument. Okay then!

Hotfixing upgrade.img is somewhat tricky, but we'll figure something out.
Comment 16 Lennart Poettering 2014-02-23 11:35:29 EST
So, what's supposed to be fixed in systemd here?
Comment 17 Will Woods 2014-02-24 09:25:04 EST
cryptsetup-generator should accept rd_LUKS_UUID like rd.luks.uuid.

Failing that, we need to work on a general plan for how to modify the system before an upgrade starts, so systemd can rewrite rd_LUKS_UUID to rd.luks.uuid.
Comment 18 Dave Mitchell 2014-03-29 20:42:27 EDT
Just as a data point, I've had similar problems upgrading from F18->F20.
It was a fresh F18 install (no previous upgrades), grub2, LVM+LUKS as set up by Anaconda. Booting from the 'System Upgrade' grub entry caused dracut to hang for a few minutes before deciding that it couldn't access any of the filesystems and dropped into the emergency shell. Adding rd.auto=1 to the boot line fixed it.

Prior to booting from the 'System Upgrade' entry, I had

# cat /etc/crypttab
luks-ffa7256c-bd83-40f6-ba54-40db40e60cf2 UUID=ffa7256c-bd83-40f6-ba54-40db40e60cf2 none 

And the grub entry was:

        linuxefi /vmlinuz-fedup root=/dev/mapper/fedora_robin-root ro rd.md=0 rd.dm=0 rd.lvm.lv=fedora_robin/root  rd.lvm.lv=fedora_robin/swap rd.luks.uuid=luks-ffa7256c-bd83-40f6-ba54-40db40e60cf2 vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8

So it *did* have the rd.luks.uuid entry, but that wasn't enough.

Hope that helps.
Comment 19 Lukáš Nykrýn 2015-05-04 11:20:09 EDT

Those options are deprecated for a looooong time.

Note You need to log in before you can comment on or make changes to this bug.