Description of problem: Upon upgrade to Fedora 15, LUKS encrypted partitions are not automatically unlocked and mounted during the boot process. Only the root partition, /, is unlocked and mounted during boot. This causes the boot process to time out, and I am dropped to an emergency shell. Version-Release number of selected component (if applicable): systemd-26-5.fc15.i686 How reproducible: Always Steps to Reproduce: 1. boot system 2. 3. Actual results: boot process times out, dropping to emergency shell Expected results: system should boot to specified runlevel Additional info: I can manually unlock each partition and mount. So as a workaround I've dropped a shell script in the root partition that I run after I've been dropped to an emergency shell. The shell script just goes through and unlocks and mounts each partition. I exit the emergency shell and the system will finish booting and works fine. System has had LUKS encrypted partitions since Fedora 11. I've upgraded through each Fedora release w/out issue until now.
Created attachment 509622 [details] fstab fstab
Created attachment 509623 [details] crypttab crypttab
The fstab shows /tmp, /var, /home, /usr, /, /opt, /var/tmp, /usr/local and swap as separate LUKS volumes. Could you please boot with "log_buf_len=1M systemd.log_level=debug systemd.log_target=kmsg" and when you get to the emergency shell, store the output of the 'dmesg' command? Attach it here (preferably as 'text/plain', not gzipped). Thanks.
Created attachment 510176 [details] dmesg dmesg from emergency shell.
I think I got the same problem, but only every fourth boot or so: after LUKS partition password request, the fedora 15 boot gets stuck and I drop into the emergency shell. Never had such an issue with 14. It is sometimes (always?) paired with the boot process asking me separately the password for all three partitions (I got a single global one), and sometimes it will then as described here fail despite me entering all three passwords and go into recovery mode/emergency shell. Sometimes it will also be just happy having asked three times and continue, or just ask once as it is supposed to and boot without trouble. All three partitions means /, /home and swap in this case. I guess it's some sort of race condition, at least on my machine here (which is an Asus Eee 1000H netbook).
Same here on a new desktop (quad core AMD) - would any additional information be useful?
I also have been seeing this both on my desktop system (quad-core Phenom 985 based) and on my netbook (Acer Aspire One 722). This is an extremely annoying problem.
I have the same problem. I believe it is some kind of timeout error. If I do anything other than speed-type the passphrase, no backspaces, then it seems to work. Anything else craps out and drops me down into emergency mode. I survived the first fedora 15 upgrade to a new kernel; that's when I first noticed it. And seconded on the annoying part.
This is having down stream repercussions as well. For instance, I had a machine where all but swap got successfully mounted on one boot. Once KDE started, there was no hibernate capability, and (probably due to a different bug) no sleep. I was able to drop to a shell and use systemd's cli tools to clear the LUKS opening problem, but KDE can't deal with that change once it's started. Most folks would need to do a full reboot and hope they get luckier next time. Can somebody please point out which part of systemd is responsible for opening LUKS volumes? I haven't found an online taxonomy of its parts yet.
in theory, this should make sure, that systemd waits for the password: add the option "comment=systemd.device-timeout=0" to the device in /etc/fstab add the option "timeout=0" to the device in /etc/crypttab
Would you please provide an example of how the added option lines should look when the file is edited (such as a before and after edit of the appropriate lines in /etc/fstab and /etc/crypttab? Is it just added to the end of each appropriate device line that is encrypted or is this on a line by itself? I would prefer to see what it should look like when it is done properly. Thanks
I tried Harald's suggestion on my laptop like so: /etc/fstab ---- before: /dev/mapper/luks-353ea87f-a852-4428-b5e4-c6346b811f46 /home ext4 defaults 1 2 after: /dev/mapper/luks-353ea87f-a852-4428-b5e4-c6346b811f46 /home ext4 comment=systemd.device-timeout=0 1 2 If you have existing options in fstab (relatime, data=journal, etc.), comma-append this. /etc/crypttab ---- before: luks-353ea87f-a852-4428-b5e4-c6346b811f46 UUID=352ea87f-a852-4428-b5e4-c6346b811f46 none after: luks-353ea87f-a852-4428-b5e4-c6346b811f46 UUID=352ea87f-a852-4428-b5e4-c6346b811f46 none timeout=0 I had no 4th 'options' column in my existing crypttab. 'man 5 crypttab' has details. I repeated this for each of the filesystems that are mounted after /, including swap. Initial report: the system boots properly. I only have this one datapoint so far.
Thanks. I made the modifications per your example. So far, so good.
The modification listed above fails. I just had to reboot, and the same issue occurred. Acer Aspire One 722, Fedora 15 64 bit, 64 Gb SSD (replaced standard HDD). I reverted the fstab and crypttab files to their original configuration, and will wait a bit more for a solution.
OK, so this is interesting: I just switched to the nvidia driver (was running nouveau), and the problem appears to have vanished. Previously, running the nouveau driver, the boot would fail 100% of the time, requiring me to employ the above mentioned workaround from the emergency shell 100% of the time. Since switching to the nvidia driver, boot has been successful 100% of the time (through about a dozen reboots thusfar), requiring no workaround. I'm scratching my head, but wondering: could Plymouth be part of the problem here? The only change during boot, not using nouveau, is that KMS isn't an option under nvidia so plymouth operates in a different mode. Could there be a problem with Plymouth's handling of the luks password in KMS mode? Let me know if you need any additional info...
I just upgraded my main workstation: http://www.smolts.org/client/show/pub_05b04b78-940b-486a-aab9-28394729f2cb to Fedora 15, and I'm seeing the problem here too, but not like I was with my laptop in the original report, but rather like the other people in this report. That is, I'm only seeing it intermittently (only 1 out of 3 reboots thus far has dropped me to the emergency console). (and I am using the nouveau driver on this machine.)
I've tried Harald Hoyer's workaround as given in Comment 10, and detailed by Bill McGonigle in Comment 12. Sadly, it didn't work on my IBM T30 laptop. The only difference I could discern was a possibly longer delay between the second prompt for the LUKS passphrase, and it's being overwritten by an (apparently unrelated) other message. The second prompt reads: Please enter passphrase for disk WDC_WD1600BEVE-11UYT0 (luks-103b57af-ec08-403a-84f2-b5f3c0531808) on /home!Started Intialize storage subsystems(RAID, LVM, etc.) where the "Started ..." is the unrelated other message. It's clear that the /home filesystem, which has the same passphrase as the others, is causing the second request. But why? If I don't enter the passphrase the second time, after a while I get: :error: unexpectedly disconnected from boot status daemon Welcome to emergency mode. ... These details are visible only if rhgb is expunged from the grub boot options, which may account for the puzzlement expressed in comments 0 and 8. I can't imagine trying to boot a system as buggy as F15 with rhgb present. Oddly, the filesystem that requires the second passphrase seems to vary, depending on the order of /etc/fstab entries. I have four encrypted filesystems: /, /home, swap, and /f14. The /boot filesystem is not encrypted. By reordering those lines, I've seen that either /home or swap is associated with the second passphrase. But I can't see any pattern or system to this outcome. Maybe it's just random. I suspect it's all due to a timing or race condition. I had reported in BZ 720178 that my IBM T30 laptop exhibits this error, while my ASUS N-10 laptop boots normally. Bill McGonigle suggests that BZ 720178 is a duplicate of this one, 716291. I concur.
I should add that the timeout workaround has greatly reduced the incidence of this symptom for me, but I still see it once in a while. Perhaps this indicates multiple code paths that can reach the same state, and some folks are seeing one more than the other.
I just came across this: http://forums.fedoraforum.org/showthread.php?t=270543 Could the partition mapping issue noted in this post have anything to do with this problem?
Just happened on my personal workstation: http://www.smolts.org/client/show/pub_3fb76057-3c12-40cb-bb36-288e8ffe22ff Upgraded it to Fedora 15 a few weeks ago. I just did a subsequent reboot and it came up fine... I've done a few reboots since upgrading to 15, so definitely an intermittent bug.
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Just got the new kernel--2.6.41.1-1 and tried to boot it up. It still fails and drops me to a dracut prompt. /home is encrypted, and it never asks for a password to unlock the partition. The last version of the kernel that will work is 2.6.38.8-35 Specs: MSI GT680R Corei7 CPU Hardware RAID 8G RAM Clean Fedora 15 install, applied all patches as they come out. This happens consistently. I have never managed to get a kernel to boot past the above version.
(In reply to comment #22) > Just got the new kernel--2.6.41.1-1 and tried to boot it up. It still fails > and drops me to a dracut prompt. > > /home is encrypted, and it never asks for a password to unlock the partition. > > The last version of the kernel that will work is 2.6.38.8-35 > > Specs: > > MSI GT680R > Corei7 CPU > Hardware RAID > 8G RAM > > Clean Fedora 15 install, applied all patches as they come out. > > This happens consistently. I have never managed to get a kernel to boot past > the above version. This seems to be another bug, if you get a dracut prompt. Please open a new bugzilla against component dracut and attach your /etc/grub.conf
Is this still an issue or can this bug be closed?
After performing some more research, I have managed to get 2.6.41.9-1 to boot consistently. The key was to remove the nouveau driver from the initramfs: su mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img dracut /boot/initramfs-$(uname -r).img $(uname -r) Substituting the kernel I wanted for uname -r while booted into the earlier kernel. Once that was accomplished, I rebooted into the 2.6.41.x kernel, reinstalled the nvidia driver (system would not complete boot without it--had to Ctrl-Alt-F2 to reach text login prompt), and reboot again. I can now boot into the 2.6.41 series kernels successfully.
Looks like this was a configuration issue/conflict between nouveau and nvidia driver.
(In reply to comment #26) > Looks like this was a configuration issue/conflict between nouveau and nvidia > driver. Can you explain why this conflict would affect LUKs? I also affected my ability to lock my screen with Ctrl-Alt-Del in XFCE. I know I solved it, but I don't know *why* the solution worked. Thanks!
(In reply to comment #26) > Looks like this was a configuration issue/conflict between nouveau and nvidia > driver. Of course it's not, don't be so jumpy to close bugs. shadowace1 isn't even the reporter and his problem doesn't look to be directly related. FWIW, this still occurs here with radeon video (not that that has anything to do with the problem). 2.6.41.8-2.5.fc15.x86_64 Somebody please re-open this bug.
this is still an issue, reopening.
I just reverted my laptop, referenced in the initial post, to the nouveau driver, and the problem returned. I had been running the nvidia driver since comment#15, and had not seen the bug during that time. After the revert, I noticed something interesting: The initial reboot after removing the nvidia driver booted fine. However, at the point I was prompted for the LUKS passphrase, Plymouth was not yet running in KMS mode. (This is because the initramfs was automatically rebuilt without the nouveau driver during the nvidia uninstall process.) After accepting my passphrase, it mounted the filesystems and began to boot, at which point it switched to KMS mode because it was able to load the nouveau driver. It continued to boot without issue. After this boot, I rebuilt the initramfs (so that nouveau is included and I get KMS immediately at boot time): mkinitrd /boot/initramfs-2.6.41.10-3.fc15.i686.img 2.6.41.10-3.fc15.i686 --force and then rebooted. At this point, I was prompted for the LUKS passphrase while Plymouth was running in KMS mode, and I'm back to where I was in the initial post (only / was unlocked and mounted, I was dropped to an emergency shell, and I have to run my script to manually unlock and mount the other partitions). This brings me back to my question that posed in comment#15: "Could there be a problem with Plymouth's handling of the LUKS passphrase while in KMS mode?" Otherwise I'm not sure why this fails constantly in KMS mode, and works constantly when not in KMS mode? Also, this is only consistently an issue on this laptop. My workstations, mentioned in comment#16 and comment#20, only see this occasionally. Other people that have commented on this but post tend to see this issue occasionally too.
Chad, are you running plymouth in text mode or graphical mode? Two kernel variables might be at play: kms (i.e nomodeset) and plymouth ( i.e. rhgb ). I'm seeing this with kms but without rhgb on my commandline. If I understand this correctly, plymouth will run either way, but without 'rhgb' plymouth will run in a text mode on kms. If there is no kms, it will run on an fb device in text mode.
Still the same here aswell. Intel KMS fedora 16. When asking for a password multiple times (often WHILE I am typing the password), Fedora decides it cannot unlock various partitions and drops me to an emergency shell. Happens sporadically but still pretty often. For me, it seems to be related with #749027 and the random password rerequests. When I am not going to the shell to avoid #708771 the whole boot hangs, I am assuming because Plymouth gets confused over its own password requests, partition unlock fails and the whole system hangs. Or maybe all of those issues are unrelated to each other. However, the whole boot is a pretty broken mess since Fedora 15. Would be about time someone attempts to repair it.
A reporter mentioned an reliable workaround in bug 749027 comment three add "plymouth.debug=file:/run/plymouth/debug.log" to the kernel command line. Can someone confirm/deny this workaround?
Bill, I am running "rhgb quiet" as a boot parameter in grub Although, I have to hit ESC (essentially turning the "quiet" part of rhgb off) shortly after entering my password in KMS mode to see the text scrolling by, else I'll never see the emergency shell prompt.
Jóhann, surprisingly that does work. I just tried it a second time on my laptop to make sure it wasn't a fluke, and twice it has booted w/out issue with nouveau and KMS. So it would seem that this really is a Plymouth issue?
Chad, could you boot once without "rhgb quiet" (and without logging it seems) to rule out graphical plymouth as being part of your problem?
Bill, I removed "rhgb quiet" (and the logging), and tried booting it that way three times, and three times straight and it failed.
...and adding logging back in, while leaving rhgb out works. so the plymouth debug logging causes this to work, strangely enough, with or without rhgb.
Adding the plymouth maintainer who might shed some lights on what's happening here...
Reassigning against plymouth afters speaking with it's maintainer. This is a fall out from requests being asynchronous now. What happens is that system needs a password so it first asks plymouthd if there are any already known passwords then plymouth will either respond with the password or will instead say "i don't know any" then it will ask plymouth to ask for a password. Now you can end up in a situation where you end up with two processes asking for any known passwords at the same time plymouthd telling both of them at the same time "i don't know any"then both processes at the same time saying "okay ask the user" At that point plymouthd will know about that password and the next request that comes by plymouthd won't say "i don't know any" instead it will reply with what it knows... The fix should be pretty straightforward and will be dealt with as soon as he has the time to do it. Thanks for your patience.
to completly disable plymouth in the initramfs add "rd.plymouth=0" to the kernel command line.
My observation is that I see this on my netbook which has an SSD drive but not on my desktop which has a standard HDD. I entered the "rd.plymouth=0" to the kernel command line. It had the Loading initial ramdisk ... Enter passphrase for /dev/sda3: I did so. Immediately after it had Enter passphrase for /dev/sda4: I did so. Then I got the luks . . . . systemd-fschk(694): /dev/sda2: clean Started wait for storage scan Starting Initialize storage subystems (2 lines) Started Show Plymouth Boot Screen (What? I thought it was supposed to be disabled!) Started Forward Password Requests to Plymouth Please enter passphrase for disk OCZ_VERTEX_PLUS (luks-.......) on /home!: I stopped at this point to enter this information. Had I entered the password, it would have finished booting. It did drop to an emergency maintenance screen, but when I hit ctrl-D, it went back to the plymouth password request box upon which I hit an escape to get back to the command line. After entering the password, the boot completed as expected. This may not be news, but I didnt see anyone previously post the boot sequence he/she was seeing. The listing is not complete. I did not give the full luks partition information. This SSD has only Fedora Linux, set up all primary partitions, /, /home, and swap encrypted, boot unencrypted, grub2 for boot manager. There was no previous operating system on it as I had just recently acquired the drive and this was an initial operating system install on it.
Harald Hoyer: I've tried your suggestion on my IBM T30 laptop that prompts twice for the LUKS password. Firstly, with "plymouth.debug=file:/run/plymouth/debug.log" added to the kernel line in /etc/grub2/grub.cfg, there is no second prompt for the LUKS password, although a huge amount of extra verbiage is spewed to the screen. Second, with that phrase removed, and "rd.plymouth=0" added by editing the kernel options manually at boot time, the huge spewage of plymouth messages is suppressed but the second prompt returns. Moreover, there's a change in behaviour: When the password is typed the first time, no '*'s are echoed, but after the second prompt, they are. Evidently there's some difference in who's asking. Third, if both phrases are used together, there are two prompts, one for /dev/sda8, the second for /dev/sda6, then the plymouth spewage, and buried in that, a third prompt - for /home. In short, the worst of all combinations. The first two don't echo; the third one does echo '*'s. I tried the third case again with different results - only two prompts. Clearly, there's a race condition somewhere. I hope these details may stimulate a glimmer of recognition of the problem. For now, I'll live with case 1, with the plymouth verbiage, and a single prompt, until a real fix comes along. Thank you.
Is Plymouth absolutely vital or can an alternative be used? If so, how would one go about switching from Plymouth to that utility? How likely would any alternative also be likely to suffer from this same problem?
(In reply to comment #42) > My observation is that I see this on my netbook which has an SSD drive but not > on my desktop which has a standard HDD. > > I entered the "rd.plymouth=0" to the kernel command line. > > It had the Loading initial ramdisk ... > Enter passphrase for /dev/sda3: > I did so. Immediately after it had > Enter passphrase for /dev/sda4: > I did so. Then I got the luks . . . . > systemd-fschk(694): /dev/sda2: clean > Started wait for storage scan > Starting Initialize storage subystems (2 lines) > Started Show Plymouth Boot Screen (What? I thought it was supposed to be > disabled!) It only disables plymouth in the initramfs. For the real system you would have to "systemctl mask" the plymouth services.
(In reply to comment #43) > Harald Hoyer: > Third, if both phrases are used together, there are two prompts, one for > /dev/sda8, the second for /dev/sda6, then the plymouth spewage, and buried in > that, a third prompt - for /home. In short, the worst of all combinations. > The first two don't echo; the third one does echo '*'s. > Well, if you disable plymouth in the initramfs, then you get the direkt prompt of "cryptsetup luksOpen", which does not echo any characters (which is a good thing in my eyes, because you cannot tell the length of the string)
A new kernel arrived today - 3.2.9-2.fc16.i686.PAE - but the race condition, apparently in plymouth, that prompts twice for the LUKS passphrase is not fixed. The workaround of including plymouth.debug=file:/run/plymouth/debug.log in the kernel boot options still works to suppress that extraneous passphrase request. The meaningless (to me) spewage that this creates is still cluttering the screen, instead of being sent ONLY to the log file. This comment is simply a note that this problem remains unsolved. I remain hopeful that it will be fixed soon.
The workaround (plymouth.debug=file:/run/plymouth/debug.log) in the kernel boot options fails miserably with my Acer Aspire One 722 netbook into which I installed an OCZ Vertex Plus+ SSD. Instead of completing, I filled in the password in the first box. After that, the Fedora bubble stayed empty, and the time it normally took to enter the password, hit the ESC key so I could see the boot process, have it pop up the second password prompt, and then complete booting had long passed. I hit ESC, but it went to a black screen. The system appeared non-responsive, so I hit CTRL-ALT-Backspace which did nothing, then CTRL-ALT-F2, which also did nothing, then hit ESC again. This time, it dropped me to the emergency prompt where I hit CTRL-D to continue, upon which it popped up the 2nd password prompt. Needless to say, the modification was the first thing I removed after I got logged-in. There's got to be a better way!
This message is a notice that Fedora 15 is now at end of life. Fedora has stopped maintaining and issuing updates for Fedora 15. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At this time, all open bugs with a Fedora 'version' of '15' have been closed as WONTFIX. (Please note: Our normal process is to give advanced warning of this occurring, but we forgot to do that. A thousand apologies.) Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, feel free to reopen this bug and simply change the 'version' to a later Fedora version. Bug Reporter: Thank you for reporting this issue and we are sorry that we were unable to fix it before Fedora 15 reached end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged to click on "Clone This Bug" (top right of this page) and open it against that version of Fedora. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
I think this bug winds up being a duplicate of bug 679907 anyway, which is still open.
It may be a moot point. I have not encountered this issue with Fedora 17.
I haven't encountered the issue with F17 either. The machine that was worst affected by this bug, and the machine in the initial bug report is still running F16. On that box, I went ahead and disabled the workaround that Jóhann suggested in comment #33, and I confirmed worked in comment #35. I have now rebooted that machine three times with that workaround removed and have not experienced this issue. So this bug may be fixed? For now, I'll assume so. If the bug rears its head again, I'll clone this report.