Red Hat Bugzilla – Bug 1012899
Upgrading to F20 with luks doesn't boot before upgrade
Last modified: 2015-05-04 11:20:09 EDT
Created attachment 803891 [details]
sos report of ramdisk
Description of problem:
On a fresh updated F19, with encrypted harddrive ( 2 partitions, 1 for /boot, 1 for encrypted system + lvm ), the initrd generated by fedup doesn't boot.
After a quick investigation, it seems to be because it doesn't detect the luks encrypted volume. Looking at the initrd generated, it doesn't have anything in /etc/cmdline.d/90crypt.conf .
I attach to this bug the rdsosreport.txt
What's in /etc/cmdline.d/90crypt.conf? What created that file?
According to the attached report, this is empty.
I do not know what create the file, maybe that's a dracut problem, so maybe this should be reassigned ?
( but I was unable to pinpoint when and how the initrd is created to further investigate)
And as it seems I forgot to explain what i did, I did run this as root :
# fedup --network 20
and then rebooted on the initrd/kernel.
Hope that answer the question
Ok, I found the problem and I have a patch :
Without /etc/crypttab, systemd do not generate the unit, and so the whole initramdisk fail to mount / and so the system cannot continue upgrading.
I can confirm that apparently, this problem is still present. When I try to reboot after completing fedup to perform the actual upgrade from F19 to F20, after entering my LUKS passphrase I land in an emergency console.
There I found some mentions in the logs of /etc/crypttab missing, so I assume it's the same problem, but don't know how to investigate it further. I'll be happy to provide more information if needed.
It seems dracut never creates a new initrd-fedup.img so Luks cannot work at fedup phase.
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 rd.luks.uuid=luks-*** $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rd.lvm.lv=vg_sys/usr rd.lvm.lv=vg_sys/root vconsole.font=latarcyrheb-sun16 rd.lvm.lv=vg_sys/swap vconsole.keymap=de"
Still no success.
fedup needs much more tests...
For me, the upgrade worked when I updated to fedup 0.8 from the upgrades-testing repo.
It does not copy /etc/crypttab into initrd-fedup.img which obviously causes that trouble.
Same issue here.
Encrypted LVM device.
After reboot I end-up in the emergency console.
I guess the initramfs is not touched at all before reboot. I guess that without cryptsetup it wont work.
# ls -l /boot/initramfs-fedup.img
-rw-r--r--. 1 root root 32346504 2013-12-12 15:05 /boot/initramfs-fedup.img
# md5sum /boot/initramfs-fedup.img
I am seeing the same thing in upgrading from F19->F20. I am seeing the same symptoms as described in the last few comments. I also noted that none of the files in /etc/cmdline.d have any content, they are all 0 length.
Would rebuilding the initramfs-fedup.img help as an interim fix?
The fedup initramfs isn't generated on your system. It's a generic initramfs that should work for any system, provided you have the necessary boot arguments.
The installer sets up these boot arguments for you, so generally /etc/crypttab isn't needed in this case - unless:
1) you set up LUKS by hand and didn't add those boot arguments, or
2) you (re)wrote grub.cfg/grub2.cfg by hand and left them out
1) Do you have a 'rd.luks.uuid=luks-<UUID>' argument in your grub.cfg/grub2.cfg for each LUKS device?
2) Are those devices listed in /etc/crypttab?
3) Does the upgrade start if you add 'rd.luks.uuid=luks-<UUID>' for everything in /etc/crypttab?
Basically, what's in your /etc/crypttab that the system needs to start up?
I didn't add luks by hand, and I didn't removed grub config to remove them. However, i carried update since I F16 or F15, so it could very well be a left over from that time.
So to answer, question 1 :
Here is what I have in grub.cfg :
title Fedora (3.12.7-300.fc20.x86_64) 20 (Heisenbug)
kernel /vmlinuz-3.12.7-300.fc20.x86_64 ro root=/dev/mapper/vg_dhcp19397-lv_root rd_LUKS_UUID=luks-a887dba5-82bb-483f-9b71-e086b41c4c2f rd_LVM_LV=vg_dhcp19397/lv_root rd_LVM_LV=vg_dhcp19397/lv_swap rd_NO_MD rd_NO_DM LANG=fr_FR.utf8 SYSFONT=latarcyrheb-sun16 KEYTABLE=fr rhgb
I do not know if rd_LUKS_UUID is
2) I do have 1 line in /etc/crypttab and this match the UUID :
luks-a887dba5-82bb-483f-9b71-e086b41c4c2f UUID=a887dba5-82bb-483f-9b71-e086b41c4c2f none
3) I cannot test, sorry, I did the upgrade already using my own patch, and it worked.
Looking a bit at my sosreport, I suspect the issue may be on systemd-cryptsetup-generator side. IE, it only understand /etc/crypttab and rd.luks.uuid, while dracut may understand more ( ie rd_LUKS_UUID ).
Since there is :
[ 6.840531] localhost dracut-initqueue: Failed to issue method call: Unit systemd-cryptsetup@luks\x2da887dba5\x2d82bb\x2d483f\x2d9b71\x2de086b41c4c2f.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-cryptsetup@luks\x2da887dba5\x2d82bb\x2d483f\x2d9b71\x2de086b41c4c2f.service' for details.
I think it was just not generated by the generator.
We can have 3 fixes:
- declare that as unsupported and deprecated
- modify systemd to make it support the old option
- modify fedup to make it fix the bootload and/or include /etc/crypttab to suit systemd
Option 2 is trivial, I will provide a untested patch, but I would be in favor of not carrying compat forever if possible, and this would only happen if we do fix configuration of people or at least warn them ( and if possible in a way they cannot miss if we cannot fix automatically, cause while I read the release notes, I likely skip a potential warning about "this is deprecated, do not use it" a few releases ago )
Created attachment 851941 [details]
add support for older option for crypsetup generator
Created attachment 851942 [details]
add support for older option for crypsetup generator
slightly fixed the commit message, to list what bugzilla I am talking about
Ah, so the problem is that systemd-cryptsetup-generator doesn't handle the old rd_LUKS_UUID argument. Okay then!
Hotfixing upgrade.img is somewhat tricky, but we'll figure something out.
So, what's supposed to be fixed in systemd here?
cryptsetup-generator should accept rd_LUKS_UUID like rd.luks.uuid.
Failing that, we need to work on a general plan for how to modify the system before an upgrade starts, so systemd can rewrite rd_LUKS_UUID to rd.luks.uuid.
Just as a data point, I've had similar problems upgrading from F18->F20.
It was a fresh F18 install (no previous upgrades), grub2, LVM+LUKS as set up by Anaconda. Booting from the 'System Upgrade' grub entry caused dracut to hang for a few minutes before deciding that it couldn't access any of the filesystems and dropped into the emergency shell. Adding rd.auto=1 to the boot line fixed it.
Prior to booting from the 'System Upgrade' entry, I had
# cat /etc/crypttab
luks-ffa7256c-bd83-40f6-ba54-40db40e60cf2 UUID=ffa7256c-bd83-40f6-ba54-40db40e60cf2 none
And the grub entry was:
linuxefi /vmlinuz-fedup root=/dev/mapper/fedora_robin-root ro rd.md=0 rd.dm=0 rd.lvm.lv=fedora_robin/root rd.lvm.lv=fedora_robin/swap rd.luks.uuid=luks-ffa7256c-bd83-40f6-ba54-40db40e60cf2 vconsole.keymap=uk rhgb quiet LANG=en_GB.UTF-8
So it *did* have the rd.luks.uuid entry, but that wasn't enough.
Hope that helps.
Those options are deprecated for a looooong time.