Description of problem: When running fedup from f21 to f22, disk is not decrypted when rebooting (password is ok, tested it several times and works on a normal boot) Version-Release number of selected component (if applicable): fedup-0.9.1-1.fc21.noarch fedup-dracut-plymouth-0.9.0-1.fc21.noarch fedup-dracut-0.9.0-1.fc21.x86_64 systemd-216-21.fc21.x86_64 How reproducible: Try to upgrade with an encrypted disk (lvm + luks) Actual results: disk not decrypted Expected results: disk decrypted and finishing upgrade Additional info:
Proposed as a Blocker for 22-beta by Fedora user jehane using the blocker tracking app because: Block update for user using luks disk encryption
Discussed at 2015-03-30 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2015-03-30/f22-blocker-review.2015-03-30-16.04.log.txt . Accepted as a blocker per criterion "For each one of the release-blocking package sets, it must be possible to successfully complete an upgrade from a fully updated installation of the previous stable Fedora release with that package set installed. The user must be made to specify which Product (or none) they wish to have running when upgrade is complete.", in the (fairly common) case of encrypted system partitions.
Does it work if you're using a US/querty keyboard layout? If so, this is probably the same as bug 1205212.
(In reply to Will Woods from comment #3) > Does it work if you're using a US/querty keyboard layout? If so, this is > probably the same as bug 1205212. On my laptop (not much info on original entry so not sure if it's the same the original submitted saw) I don't even get a password prompt. After a timeout period it drops to emergency maintenance saying it cannot find /dev/disk/by-id/<UUIDHERE> ... and a look at /dev/disk/by-id indeed shows that missing. Looking around the initial fedup environment quickly I don't see /etc/crypttab or cryptsetup ... could it be these were missed somehow in building the fedup environment and consequently it never has the config to find and unlock the encrypted filesystems? I'm on 3G internet only at the moment so it's tricky doing full builds and updates to reproduce ...
Yes, disk is decrypted is I use a US/qwerty layout. My laptop is configured to use french azerty layout by default. But it's now hanging after the "switching roo"
Hmm perusing old bugs revealed this one: https://bugzilla.redhat.com/show_bug.cgi?id=1012899 Which is very much like what I'm seeing trying to go from F21 to F22 Workstation product (on Beta TC5). My kernel command line is: linuxefi /vmlinuz-fedup root=UUID=0f1d4798-1e9c-4b25-9ae2-2a0de5ac2284 ro rootflags=subvol=root vconsole.font=latarcyrheb-sun16 rd.luks.uuid=luks-07625f19-11fd-4537-9c42-a5a8b1732277 rd.luks.uuid=luks-cbd76143-205b-463e-8efd-29b9e33d7b59 rhgb quiet zswap.enabled=1 zswap.zpool=zsmalloc LANG=en_GB.UTF-8 If I take out rhgb and quiet and set systemd.log_level=debug I see it spinning on "settle" for a long while before declaring it can't find disk 0f1d4798-1e9c-4b25-9ae2-2a0de5ac2284 ... note that I never get a luks password prompt.
Okay digging around the fedup initrd environment ... These days what is meant to be responsible for decrypting the disks? There is no dracut /etc/cmdline.d snippet for cryptsetup and there is no cryptsetup generator in /usr/lib/systemd/system-generators in the environment either ... I suspect crossed wires? It appears nothing is paying attention in the F22 Beta TC5 fedup initrd environment to actually note the luks encrypted volume and decrypt it ready for mounting and use.
(In reply to marianne from comment #5) > Yes, disk is decrypted is I use a US/qwerty layout. > My laptop is configured to use french azerty layout by default. > > But it's now hanging after the "switching roo" Ah interesting so you do get a password prompt? The switching root is a known issue BZ1185604 ... You need to update your *F21* systemd with the version that's in updates-testing right now (or at least still was this morning)... The minimum version to fix that is systemd-216-24.fc21
(In reply to marianne from comment #5) > Yes, disk is decrypted is I use a US/qwerty layout. > My laptop is configured to use french azerty layout by default. Yep, that sounds like bug 1205212. How did you configure your system to use the azerty layout? Did you set that up in the installer, or did you manually set that up? (e.g. by using localectl, or editing /etc/vconsole.conf, or something like that...) > But it's now hanging after the "switching roo" As James points out in comment #8, if you haven't updated to systemd-216-24.fc21 (or newer) you'll hit bug 1185604.
(In reply to Will Woods from comment #9) > (In reply to marianne from comment #5) > > Yes, disk is decrypted is I use a US/qwerty layout. > > My laptop is configured to use french azerty layout by default. > > Yep, that sounds like bug 1205212. > > How did you configure your system to use the azerty layout? Did you set that > up in the installer, or did you manually set that up? (e.g. by using > localectl, or editing /etc/vconsole.conf, or something like that...) > I set this up in the installer. Everything was working fine. > > But it's now hanging after the "switching roo" > > As James points out in comment #8, if you haven't updated to > systemd-216-24.fc21 (or newer) you'll hit bug 1185604. Works for me (well, graphic display is broken since fedup but it's an other bug ...)
On the confusion between James' case and Marianne's: when you use an upgrade.img from Beta TC2 or newer, the decrypt prompt is not even shown, as James says. That is being tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1208214 . I'm guessing Marianne is simply running 'fedup --network 22', which will result in the Alpha upgrade.img being used. That one does not have the bug James is encountering, but it sounds like it does have a keyboard layout issue.
As of TC7/TC8, James' bug should be fixed, so we should be able to test this scenario with the most recent bits. I will confirm whether this bug is still present.
I can't reproduce this myself. Test procedure: Minimal install of F21 Final using Server network install image, encrypt data with passphrase 'vrqi,entchevql' (that's what you get by typing 'vraimentcheval' on a US keyboard with the French layout selected). Upgrade with fedup --network 22 --instrepo https://dl.fedoraproject.org/pub/alt/stage/22_Beta_TC8/Server/x86_64/os . That worked fine for me: entering the passphrase when booting into fedup env worked correctly, and entering the passphrase for the upgraded system also worked correctly.
*** Bug 1205212 has been marked as a duplicate of this bug. ***
I have tried this and can't reproduce it either. I used F21 Workstation installation, configured anaconda to use French keyboard and encrypted the disk with "azerty" (written as "qwerty" on US keyboard). After that I fully updated the system and ran fedup with Beta RC1 upgrade.img. Fedup unlocked the disk correctly and the upgraded system also unlocks the disk correctly. The french keyboard is still set in the system.
As written on https://bugzilla.redhat.com/show_bug.cgi?id=1205212#c6 a /etc/vconsole.conf file is not created in the fedup initramfs. This might be part of the problem.
But, as we said, we are testing and not seeing the bug at all. It works as intended. Are you using the upgrade.img from Beta TC8 or later, using the --instrepo parameter as shown in #c13?
I repeated the steps as of comment #13 (but with workstation instead of server since I just have a workstation installation at hand). Still the same issue: no /etc/vconsole.conf in initramfs according to lsinitramfs.
(In reply to Christian Stadelmann from comment #18) > I repeated the steps as of comment #13 (but with workstation instead of > server since I just have a workstation installation at hand). Still the same > issue: no /etc/vconsole.conf in initramfs according to lsinitramfs. Ugh, okay, I might have an idea what's happening here. Newer dracut uses an early_cpio to load microcode before doing anything else (maybe not on every system, but definitely on mine). This is a tiny, uncompressed initramfs that's stuck onto the *front* of a normal initramfs. The kernel will happily let you concatenate initramfs images - even if they have totally different compression types - and it will unpack them just fine. But most userspace tools don't handle this. For example, if I examine my initramfs manually: [wwoods@metroid ~]$ img=/boot/initramfs-$(uname -r).img [wwoods@metroid ~]$ file $img /boot/initramfs-3.19.3-200.fc21.x86_64.img: ASCII cpio archive (SVR4 with no CRC) [wwoods@metroid ~]$ cat $img | cpio -t . kernel kernel/x86 kernel/x86/microcode kernel/x86/microcode/GenuineIntel.bin early_cpio 26 blocks Notice how there's no /etc/vconsole.conf listed - we can only see the early_cpio. But lsinitrd sees it just fine: [wwoods@metroid ~]$ lsinitrd $img | grep vconsole.conf -rw-r--r-- 1 root root 37 Oct 8 2014 etc/vconsole.conf It turns out dracut ships a 'skipcpio' tool for this exact purpose: [wwoods@metroid ~]$ /usr/lib/dracut/skipcpio $img > real.img [wwoods@metroid ~]$ file real.img real.img: gzip compressed data, last modified: Thu Apr 2 14:33:17 2015, max compression, from Unix [wwoods@metroid ~]$ gzip -dc real.img | cpio -t | grep vconsole.conf etc/vconsole.conf 74060 blocks So: either I need to fix fedup to understand early_cpio, or I should make fedup just use 'lsinitrd' to read files out of initramfs.
Discussed at today's blocker review meeting [1]. Resolution of discussion: We have a better idea of a wrinkle in initramfs layout which could cause #1207251 to occur in some cases but not others; the fix would be entirely on the from-release side and so does not need to block F22 image compose. wwoods aims to have a fix ready by Thursday. [1] http://meetbot.fedoraproject.org/fedora-blocker-review/2015-04-13/
To join up a couple of dots here for future reference, if I'm understanding correctly: fedup, when running on the 'source' distribution, grabs the generic upgrade.img file from the 'target' release. Then it reads several files from the initramfs for the currently-booted kernel and gloms them into upgrade.img , producing a custom upgrade initramfs for the system being upgraded. The theory is that fedup fails when it tries to read the source files if the initramfs for the currently-booted kernel has the microcode chicanery in it. Thus it's possible that whether you see this bug depends on the kernel you're running on the 'source' distribution, the dracut version used to build the initramfs, and even possibly on the system's CPU (I haven't checked, but it may perhaps be the case that we only do the early-microcode-application stuff when we think it's 'needed').
(In reply to Will Woods from comment #19) > Notice how there's no /etc/vconsole.conf listed - we can only see the > early_cpio. But lsinitrd sees it just fine: Will, I have no idea why it works for you, but I remember Harald Hoyer mentioning several times that lsinitrd can't display concatenated initrds properly. We even have it mentioned in one of our test cases, per his advice: https://fedoraproject.org/wiki/QA:Testcase_Kickstart_File_Path_Ks_Cfg So before you completely rely on lsinitrd, please consult with Harald whether lsinitrd has been improved lately, or whether there are cases where this tool will fail you as well.
(In reply to Kamil Páral from comment #22) > (In reply to Will Woods from comment #19) > > Notice how there's no /etc/vconsole.conf listed - we can only see the > > early_cpio. But lsinitrd sees it just fine: > > Will, I have no idea why it works for you, but I remember Harald Hoyer > mentioning several times that lsinitrd can't display concatenated initrds > properly. We even have it mentioned in one of our test cases, per his advice: > https://fedoraproject.org/wiki/QA:Testcase_Kickstart_File_Path_Ks_Cfg > > So before you completely rely on lsinitrd, please consult with Harald > whether lsinitrd has been improved lately, or whether there are cases where > this tool will fail you as well. True, it doesn't handle *arbitrarily* concatenated initramfs images. (There's no userspace tool that currently does.) But dracut will *only* generate images that lsinitrd *can* read (and, therefore, find etc/vconsole.conf inside). So lsinitrd will work for the common case. For custom cases: if you generate your own arbitrary-concatenated initramfs images using non-standard tools, lsinitrd will still work *if* the image that contains all the system config files is first - or second, after an early_cpio image. This should be fine: everything I've ever seen (tools and docs included) recommends *appending* new images to the *existing* initrd. So: lsinitrd will work for the standard case, and for all custom cases that I've ever heard of. If you need to get *really* weird, well, maybe you ought to write that userspace tool that unpacks arbitrary-concatenated initramfs images, so I can make upgrades work for you?
Will: it may have changed recently, but I certainly recall when I first came across this whole microcode thing, lsinitrd wouldn't show the contents of the concatenated initramfs properly, it only showed the early-init stuff. Not any kind of crazy custom initramfs, just a stock Fedora one. If it's been fixed, though, great.
http://koji.fedoraproject.org/koji/buildinfo?buildID=628917 ought to fix this bug: can people please test with that version of fedup? Thanks!
fedup-0.9.2-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/fedup-0.9.2-1.fc21
fedup-0.9.2-1.fc20 has been submitted as an update for Fedora 20. https://admin.fedoraproject.org/updates/fedup-0.9.2-1.fc20
fedup-0.9.2-1.fc22 has been submitted as an update for Fedora 22. https://admin.fedoraproject.org/updates/fedup-0.9.2-1.fc22
Package fedup-0.9.2-1.fc20: * should fix your issue, * was pushed to the Fedora 20 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing fedup-0.9.2-1.fc20' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-6467/fedup-0.9.2-1.fc20 then log in and leave karma (feedback).
*** Bug 1214202 has been marked as a duplicate of this bug. ***
No change. fedup still does not put a /etc/vconsole.conf inside initramfs-fedup.img according to lsinitrd.
fedup-0.9.2-1.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report.
fedup-0.9.2-1.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
Re-opening per #c31, but I suspect we may need more info from Christian, perhaps even the actual initramfs files. I still have not been able to reproduce this no matter how I twiddle with the F21 install.
I've sent the file to both Adam Williamson and Will Woods. Any more information needed?
I run fedup again and it worked without problems. I don't know why.
Re-closing, then. Thanks for retesting.
*** Bug 1224985 has been marked as a duplicate of this bug. ***
My error, sorry