Bug 543236 - No root device found on encrypted root at first reboot after long uptime
Summary: No root device found on encrypted root at first reboot after long uptime
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: plymouth
Version: 12
Hardware: All
OS: Linux
medium
high
Target Milestone: ---
Assignee: Ray Strode [halfline]
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 551250 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-12-01 23:54 UTC by Michael Ploujnikov
Modified: 2010-02-09 23:01 UTC (History)
7 users (show)

Fixed In Version: 0.8.0-0.2009.29.09.19.3.fc12
Clone Of:
Environment:
Last Closed: 2010-01-30 04:29:29 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
init.log (26.40 KB, text/plain)
2009-12-01 23:54 UTC, Michael Ploujnikov
no flags Details
dmesg (42.27 KB, text/plain)
2009-12-02 00:30 UTC, Michael Ploujnikov
no flags Details
grub menu.lst (753 bytes, text/plain)
2009-12-02 00:31 UTC, Michael Ploujnikov
no flags Details
initrd (10.89 MB, application/octet-stream)
2010-01-06 18:02 UTC, Michael Ploujnikov
no flags Details
newest initrd (10.89 MB, application/octet-stream)
2010-01-23 16:23 UTC, Michael Ploujnikov
no flags Details

Description Michael Ploujnikov 2009-12-01 23:54:29 UTC
Created attachment 375243 [details]
init.log

Description of problem:
I installed F12 on my thinkpad t500 laptop the week it F12 came out. I chose to encrypt the whole root disk during installation. Since then I kept the laptop running all the time and resulting in a high uptime by putting it in sleep mode during the night hours and charging the battery during the day. Unfortunately, it happend that last night the battery didn't last through the whole night even in suspend mode so I had to reboot the laptop. Once I rebooted the laptop F12 failed to boot with only the following messages appearing on the screen:

"
plymouthd: ply-keyboard.c:450: ply_keyboard_add_input_handler: Assertion 'keyboard != ((void *)0)' failed.


No root device found

Boot has failed, sleeping forever.
"

My first reaction was to check if the encrypted root somehow got corrupted due to powerloss. I booted a LiveUSB and was successfully able to decrypt the root disk, find the LVM volume group and mount the root volume. During this first mount after the reboot, dmesg showed some FS errors about missing/invalid/wrong references (I forgot to write down the message exactly).

Booting with "rhgb quiet" removed and "rdshell" added to the grub kernel arguments gives me these message (typed by hand as best as I could):

"
[drm] LVDS-8: set mode 1280x800 c
Console: switching to colour frame buffer device 160x50
fb0: inteldrmfb frame buffer device
registered panic notifier
acpi device:03: registered as cooling_device2
input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/device:02/input/input6
ACPI: Video Device [VID] (multi-head: yes  rom: no  post: no)
[drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
dracut: Starting plymouth daemon
dracut: Starting plymouth daemon
plymouthd: ply-keyboard.c:450: ply_keyboard_add_input_handler: Assertion 'keyboard != ((void *)0)' failed.
dracut: error: unexpectedly disconnected from boot status daemon
dracut: error: unexpectedly disconnected from boot status daemon
dracut: rd_NO_MDIMSM: no MD RAID for imsm/isw raids
dracut: rd_NO_MDIMSM: no MD RAID for imsm/isw raids
yenta_cardbus: 0000:15:00.0: CardBus bridge found [17aa:20c6]
yenta_cardbus: 0000:15:00.0: ISA IRQ mask 0x0cb8, PCI irq 16
yenta_cardbus: 0000:15:00.0: Socket status: 30000006
yenta_cardbus: 0000:15:00.0: pcmcia: parent PCI bridge I/O window: 0x4000 - 0x7fff
yenta_cardbus: 0000:15:00.0: pcmcia: parent PCI bridge Memory window: 0xf4800000 - 0xf7ffffff
yenta_cardbus: 0000:15:00.0: pcmcia: parent PCI bridge Memory window: 0xf0000000 - 0xf3ffffff
dracut: luksOpen /dev/sda3 luks-cbb32109-3b6d-433b-8da8-0e7b4f9387a7
dracut: luksOpen /dev/sda3 luks-cbb32109-3b6d-433b-8da8-0e7b4f9387a7
IBM TrackPoint firmware: 0x0e, buttons: 3/3
input: TPPS/2 IBM TrackPoint as /devices/platform/i8042/serio1/serio2/input/input7
dracut: Scanning devices sda2  for LVM volume groups
dracut: Scanning devices sda2  for LVM volume groups
dracut Reading all physical volumes. This may take a while...
dracut Reading all physical volumes. This may take a while...


No root device found

Dropping to debug shell.

sh: can't access tty; job control turned off
# _
"
(Yes, a lot of those lines did appear to be duplicated on my screen. I also noticed that with rdinitdebug kernel argument these messages don't appear to be duplicated)

Booting with "rhgb vga=0x318" as directed here doesn't fix the problem: http://fedoraproject.org/wiki/Bugs/Common#Encrypted_disks_can.27t_be_unencrypted_for_non-graphical_boot

Version-Release number of selected component (if applicable):


How reproducible:
With the current state of the F12 OS on my laptop: always.

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

# blkid
/dev/sda1: UUID="7b998344-d01b-4924-918a-eacbe7f61a9e" TYPE="ext4" 
/dev/sda2: UUID="B5z730-ayyC-Xzm8-7EcC-l9Sd-8k3m-U1uvIG" TYPE="LVM2_member" 
/dev/sda3: UUID="cbb32109-3b6d-433b-8da8-0e7b4f9387a7" TYPE="crypto_LUKS" 
/dev/sdb1: LABEL="FEDORA" UUID="07F9-CA66" TYPE="vfat" 
# blkid -o udev
ID_FS_UUID=7b998344-d01b-4924-918a-eacbe7f61a9e
ID_FS_UUID_ENC=7b998344-d01b-4924-918a-eacbe7f61a9e
ID_FS_TYPE=ext4
ID_FS_UUID=B5z730-ayyC-Xzm8-7EcC-l9Sd-8k3m-U1uvIG
ID_FS_UUID_ENC=B5z730-ayyC-Xzm8-7EcC-l9Sd-8k3m-U1uvIG
ID_FS_TYPE=LVM2_member
ID_FS_UUID=cbb32109-3b6d-433b-8da8-0e7b4f9387a7
ID_FS_UUID_ENC=cbb32109-3b6d-433b-8da8-0e7b4f9387a7
ID_FS_TYPE=crypto_LUKS
ID_FS_LABEL=FEDORA
ID_FS_LABEL_ENC=FEDORA
ID_FS_UUID=07F9-CA66
ID_FS_UUID_ENC=07F9-CA66
ID_FS_TYPE=vfat
# dmsetup ls --tree
No devices found
# cat /proc/cmdline
ro root=/dev/mapper/vg_f12-lv_root noiswmd LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us init=/sbin/bootchartd rdshell

(/dev/sdb1 is a usb stick I used to transfer the logs from the laptop)

I selected high severity because this problem has become a serious productivity killer since I can't use my laptop to do my work and I would like to fix this problem rather than re-installing F12 (my data is safely backed up).

Comment 1 Michael Ploujnikov 2009-12-02 00:30:59 UTC
Created attachment 375253 [details]
dmesg

Comment 2 Michael Ploujnikov 2009-12-02 00:31:23 UTC
Created attachment 375255 [details]
grub menu.lst

Comment 3 Michael Ploujnikov 2009-12-02 02:47:47 UTC
Just out of curiosity I tried running:

/bin/plymouth ask-for-password --prompt /dev/sda3 is password protected --command=/sbin/cryptsetup luksOpen -T1 /dev/sda3 luks-cbb32109-3b6d-433b-8da8-0e7b4f9387a7; echo $?

and got 111 as a result. This makes me think that the reason my encrypted root isn't mounted is because the passkey prompt fails to show up. Why that happens, I'm not yet sure.

Comment 4 Michael Ploujnikov 2009-12-02 13:56:28 UTC
Based on the /var/log/yum.log that I found using the liveUSB I have plymouth-0.8.0-0.2009.29.09.19.fc12.x86_64 installed on this laptop.

Comment 5 Michael Ploujnikov 2009-12-02 14:22:25 UTC
I just ran /sbin/cryptsetup luksOpen -T1 /dev/sda3 luks-cbb32109-3b6d-433b-8da8-0e7b4f9387a7; echo $? and got:

Enter passphrase for /dev/sda3:
padlock: VIA PadLock not detected.
Intel AES-NI instructions are not detected.
Key slot 0 unlocked.
0

and luks-cbb32109-3b6d-433b-8da8-0e7b4f9387a7 appeared in /dev/mapper

leaving the shell (with ctrl-D) caused dracut to continue booting the system

Comment 6 Michael Ploujnikov 2009-12-19 15:31:35 UTC
PING

I would like someone to respond to this issue please. I haven't restarted my laptop since reporting this issue for fear of running into it again. I have about 300 updates outstanding because I don't want to change the state of my system too much.

Comment 7 Michael Ploujnikov 2009-12-22 18:45:02 UTC
Re-assigning to plymouth because this doesn't seem like a problem with dracut (I can boot by issuing commands manually) and to get some response about the problem.

Comment 8 Ray Strode [halfline] 2010-01-05 15:41:09 UTC
duplicated output messages 
and 
"plymouthd: ply-keyboard.c:450: ply_keyboard_add_input_handler: Assertion
'keyboard != ((void *)0)' failed."

are two separate issues that were fixed before F12 GA.  Were you running rawhide before upgrading to F12?  If so, you may need to rebuild your initrd with the GA plymouth.

Comment 9 Michael Ploujnikov 2010-01-05 16:51:46 UTC
I did not use rawhide. I installed F12 from scratch. How can I rebuild the initrd? When is the initrd rebuilt normally? Does it happen when plymouth is updated? I should try checking if there is a plymouth/dracut update applicable for my system. What should I do in case all of these problems go away after the initrd is rebuilt?

Comment 10 Ray Strode [halfline] 2010-01-05 17:25:52 UTC
hmm, if this is a fresh install then there must be a lingering issue.

The initrd is rebuilt when doing a new kernel install.

Comment 11 Ray Strode [halfline] 2010-01-05 17:26:42 UTC
*** Bug 551250 has been marked as a duplicate of this bug. ***

Comment 12 Michael Ploujnikov 2010-01-05 23:56:13 UTC
I updated my kernel to kernel-2.6.31.9-174.fc12.x86_64.rpm (and kernel-firmware-2.6.31.9-174.fc12.noarch.rpm) and both the main error (no root device) as well as the assertion message are still happening. I made sure that grub is selecting the 2.6.31.9* kernel when booting.

I now have this initrd:
md5sum /boot/initramfs-2.6.31.9-174.fc12.x86_64.img
b52cf2535e0d13b02af11ba792514377  /boot/initramfs-2.6.31.9-174.fc12.x86_64.img

Comment 13 Ray Strode [halfline] 2010-01-06 15:31:34 UTC
what's the output of

rpm -q plymouth

?

Comment 14 Michael Ploujnikov 2010-01-06 16:30:31 UTC
plymouth-0.8.0-0.2009.29.09.19.fc12.x86_64

Comment 15 Ray Strode [halfline] 2010-01-06 17:47:14 UTC
can you attach your initrd ?

Comment 16 Michael Ploujnikov 2010-01-06 18:02:48 UTC
Created attachment 382041 [details]
initrd

Comment 17 Ray Strode [halfline] 2010-01-06 18:09:51 UTC
thanks.

Comment 18 Michael Ploujnikov 2010-01-23 16:05:48 UTC
I've just updated to the following packages and I still get all of the problems mentioned in this bug. I still have to drop into the 'rdshell' in order to enter the luks encryption key and boot my laptop:

Jan 22 17:50:55 Updated: kernel-firmware-2.6.31.12-174.2.3.fc12.noarch
Jan 22 17:51:53 Installed: kernel-2.6.31.12-174.2.3.fc12.x86_64
Jan 23 10:56:40 Updated: libdrm-2.4.17-1.fc12.x86_64
Jan 23 10:56:42 Updated: libdrm-2.4.17-1.fc12.i686
Jan 23 10:56:44 Updated: plymouth-0.8.0-0.2009.29.09.19.1.fc12.x86_64
Jan 23 10:56:44 Updated: plymouth-utils-0.8.0-0.2009.29.09.19.1.fc12.x86_64
Jan 23 10:56:45 Updated: plymouth-gdm-hooks-0.8.0-0.2009.29.09.19.1.fc12.x86_64
Jan 23 10:56:47 Updated: libdrm-devel-2.4.17-1.fc12.x86_64

Comment 19 Michael Ploujnikov 2010-01-23 16:23:37 UTC
Created attachment 386346 [details]
newest initrd

Comment 20 Ray Strode [halfline] 2010-01-25 18:34:20 UTC
ah i missed before that you're doing:

init=/sbin/bootchartd

on your kernel command line.

Comment 21 Michael Ploujnikov 2010-01-26 00:30:24 UTC
(In reply to comment #20)
> ah i missed before that you're doing:
> 
> init=/sbin/bootchartd
> 
> on your kernel command line.    

I just tried removing this from the kernel line(s) and all of the sudden I was able to boot properly into my laptop.

Comment 22 Ray Strode [halfline] 2010-01-26 18:01:04 UTC
I built 0.8.0-0.2009.29.09.19.3 that should help with this. mind trying?

Comment 23 Fedora Update System 2010-01-26 18:02:16 UTC
plymouth-0.8.0-0.2009.29.09.19.3.fc12 has been submitted as an update for Fedora 12.
http://admin.fedoraproject.org/updates/plymouth-0.8.0-0.2009.29.09.19.3.fc12

Comment 24 Michael Ploujnikov 2010-01-26 19:02:35 UTC
I'll try this tonight. Which packages should I update? All of them?

Comment 25 Ray Strode [halfline] 2010-01-27 15:04:59 UTC
all the ones you currently have installed anyway.

Also you'll need to run

dracut -f

to rebuild your initrd with the new plymouth

(as a safety measure, you may want to make sure you have multiple kernels installed or back up your existing initrd and make it an alternate entry in grub.conf)

Comment 26 Fedora Update System 2010-01-28 00:57:17 UTC
plymouth-0.8.0-0.2009.29.09.19.3.fc12 has been pushed to the Fedora 12 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update plymouth'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/F12/FEDORA-2010-1148

Comment 27 Michael Ploujnikov 2010-01-30 04:14:01 UTC
As I said on http://admin.fedoraproject.org/updates/F12/FEDORA-2010-1148, installing plymouth-0.8.0-0.2009.29.09.19.3.fc12 from updates-testing didn't fix the problem for me even after I re-generated the initramfs image.

Comment 28 Michael Ploujnikov 2010-01-30 04:29:29 UTC
After figuring out the problem with 'dracut -f' I guess that this does 'fix' the bug, but it's not pretty. If I still have init=/sbin/bootchartd then I get a console prompt for the encryption key. I'm just going to remove bootchart from my system for an easy fix.

Comment 29 Ray Strode [halfline] 2010-02-01 03:09:48 UTC
ah sorry for the misinformation.  You can also run 

/usr/libexec/plymouth/plymouth-update-initrd


plymouth by default won't show a graphical splash if init= is on the cmdline.  this is so plymouth doesn't interfere with emergency shells and the like.

We could probably special case bootchart.

You can force a graphical splash by putting plymouth:force-splash on the kernel command line.

Comment 30 Fedora Update System 2010-02-09 05:07:51 UTC
plymouth-0.8.0-0.2009.29.09.19.3.fc12 has been pushed to the Fedora 12 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.