Bug 462116

Summary: f9 to rawhide kernel upgrade chokes on encrypted volume
Product: [Fedora] Fedora Reporter: Michael J. Chudobiak <mjc>
Component: mkinitrdAssignee: Peter Jones <pjones>
Status: CLOSED RAWHIDE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: medium    
Version: rawhideCC: dcantrell, fedoraproject, notting, wtogami
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-11-04 19:57:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 438943    
Attachments:
Description Flags
Screenshot of boot failure with lvm+luks
none
log of mkinitrd output in chrooted rescue mode
none
screenshot of /proc and /sys in chrooted rescue
none
The init file, as requested none

Description Michael J. Chudobiak 2008-09-12 18:51:40 UTC
I enabled the rawhide repo and ran "yum upgrade kernel", which pulled in these rpms:

[root@xena ~]# rpm -qa | grep fc10
plymouth-0.6.0-0.2008.09.10.1.fc10.x86_64
plymouth-libs-0.6.0-0.2008.09.10.1.fc10.x86_64
libbdevid-python-6.0.63-1.fc10.x86_64
mkinitrd-6.0.63-1.fc10.x86_64
plymouth-plugin-label-0.6.0-0.2008.09.10.1.fc10.x86_64
fedora-logos-9.99.3-1.fc10.noarch
kernel-2.6.27-0.322.rc6.fc10.x86_64
kernel-firmware-2.6.27-0.322.rc6.fc10.noarch
nash-6.0.63-1.fc10.x86_64
plymouth-plugin-spinfinity-0.6.0-0.2008.09.10.1.fc10.x86_64

However, my system will no longer boot. I get messages like:

No volume groups found
Unable to access resume device
switchroot: mount failed

There is no luks passphrase prompt.

Reverting to the f9 kernel makes it boot again, and the passphrase prompt occurs normally.

The disk has lvm2 volumes with f9-anaconda installed encryption.

- Mike

Comment 1 Bill Nottingham 2008-10-27 20:35:33 UTC
Can you post a full boot log (including any kernel messages?)

What happens if you re-run mkinitrd?

Comment 2 Michael J. Chudobiak 2008-10-28 13:07:08 UTC
Bill,

1) "/" is mounted on the luks-encrypted lvm volume, so a boot log wouldn't be generated in /var/log since that volume is the one that is not mounting.

So... would I have to capture the terminal boot messages using a serial console, or am I missing something?

2) How do I re-run mkinitrd, exactly?

- Mike

Comment 3 Bill Nottingham 2008-10-28 22:54:18 UTC
(In reply to comment #2)
> So... would I have to capture the terminal boot messages using a serial
> console, or am I missing something?

Correct. Digital camera picture also could work.

> 2) How do I re-run mkinitrd, exactly?

As root:

/sbin/mkinitrd -f /boot/initrd-2.6.27-0.322.rc6.fc10.x86_64.img 2.6.27-0.322.rc6.fc10.x86_64

(You can substitute a different kernel version and file name if you have a newer one.)

Comment 4 Michael J. Chudobiak 2008-10-29 13:32:29 UTC
Created attachment 321805 [details]
Screenshot of boot failure with lvm+luks

Comment 5 Michael J. Chudobiak 2008-10-29 13:36:03 UTC
Some config info:

[mjc@xena Desktop]$ ll /dev/mapper/
total 0
crw-rw---- 1 root root  10, 60 2008-10-29 09:22 control
brw-rw---- 1 root disk 253,  0 2008-10-29 09:23 luks-sda2
brw-rw---- 1 root disk 253,  1 2008-10-29 09:23 xena_vg00-LogVol00
brw-rw---- 1 root disk 253,  2 2008-10-29 09:23 xena_vg00-LogVol01

[mjc@xena Desktop]$ more /etc/fstab
/dev/xena_vg00/LogVol00 /                       ext3	defaults,noatime	1 1
UUID=636f85fa-0a45-4f77-9e0e-3272bd2b9527 /boot                   ext3	defaults,noatime	1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/xena_vg00/LogVol01 swap                    swap    defaults        0 0
192.168.0.3:/ /fileserver nfs4 intr,suid,dev,soft,exec 0 0

[mjc@xena Desktop]$ more /etc/crypttab 
luks-sda2               /dev/sda2       none

Let me know if anything else would be useful.

Comment 6 Michael J. Chudobiak 2008-10-29 14:47:16 UTC
I booted into rescue mode and:

chroot /mnt/sysimage
mkinitrd -v -f /boot/initrd-2.6.27.4-58.fc10.x86_64.img 2.6.27.4-58.fc10.x86_64

and rebooted.

The same errors occurred. No difference.

- Mike

Comment 7 Bill Nottingham 2008-10-29 15:56:45 UTC
Did it put any actual modules in your initrd?

Comment 8 Michael J. Chudobiak 2008-10-29 16:42:16 UTC
I'm not really familiar with the boot process. How do I tell if it "put any actual modules in your initrd"?

- Mike

Comment 9 Bill Nottingham 2008-10-29 17:14:57 UTC
Can you post the full output of 'mkinitrd -v -f'... (just do 2>&1 | tee mkinitrd.log)

Comment 10 Michael J. Chudobiak 2008-10-29 18:05:21 UTC
Created attachment 321838 [details]
log of mkinitrd output in chrooted rescue mode

As requested, here is the mkinitrd output.

Comment 11 Bill Nottingham 2008-10-29 18:31:49 UTC
Wow, no modules at all. Is both /proc and /sys mounted in your rescue chroot? Is the kernel installed correctly?

Comment 12 Michael J. Chudobiak 2008-10-29 18:49:27 UTC
I'll check the /proc and /sys in rescue/chroot later.

The kernel seemed to install OK, with one plymouth-related warning. Maybe there is a missing dependency?

[root@xena ~]# yum upgrade kernel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * livna: ftp-stud.fht-esslingen.de
 * fedora: less.cogeco.net
 * updates-newkey: less.cogeco.net
 * Avtech: server2.domain.avtechpulse.com
 * rawhide: ftp.linux.ncsu.edu
 * updates: less.cogeco.net
rawhide                                                  | 2.7 kB     00:00     
553122f18bedd1c7cb673bbd361467a98800e67e-primary.sqlite. | 8.1 MB     00:48     
Setting up Upgrade Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.27.4-58.fc10 set to be installed
--> Processing Dependency: mkinitrd >= 6.0.61-1 for package: kernel
--> Processing Dependency: kernel-firmware >= 2.6.27.4-58.fc10 for package: kernel
--> Running transaction check
---> Package mkinitrd.x86_64 0:6.0.68-1.fc10 set to be updated
--> Processing Dependency: nash = 6.0.68-1.fc10 for package: mkinitrd
--> Processing Dependency: plymouth >= 0.6.0-0.2008.09.10.1 for package: mkinitrd
--> Processing Dependency: libnash.so.6.0.68()(64bit) for package: mkinitrd
--> Processing Dependency: libbdevid.so.6.0.68()(64bit) for package: mkinitrd
---> Package kernel-firmware.noarch 0:2.6.27.4-58.fc10 set to be updated
--> Running transaction check
--> Processing Dependency: libbdevid.so.6.0.52()(64bit) for package: libbdevid-python
--> Processing Dependency: nash = 6.0.52-2.fc9 for package: libbdevid-python
---> Package nash.x86_64 0:6.0.68-1.fc10 set to be updated
---> Package plymouth.x86_64 0:0.6.0-0.2008.10.27.5.fc10 set to be updated
--> Processing Dependency: system-logos >= 9.0.1 for package: plymouth
--> Processing Dependency: system-plymouth-plugin >= 0.6.0-0.2008.10.27.5.fc10 for package: plymouth
--> Processing Dependency: initscripts >= 8.83-1 for package: plymouth
--> Processing Dependency: plymouth-scripts for package: plymouth
--> Processing Dependency: libplybootsplash.so.2()(64bit) for package: plymouth
--> Processing Dependency: libply.so.2()(64bit) for package: plymouth
--> Running transaction check
---> Package fedora-logos.noarch 0:10.0.0-2.fc10 set to be updated
---> Package plymouth-scripts.x86_64 0:0.6.0-0.2008.10.27.5.fc10 set to be updated
---> Package plymouth-libs.x86_64 0:0.6.0-0.2008.10.27.5.fc10 set to be updated
---> Package initscripts.x86_64 0:8.84-1 set to be updated
---> Package libbdevid-python.x86_64 0:6.0.68-1.fc10 set to be updated
---> Package plymouth-plugin-solar.x86_64 0:0.6.0-0.2008.10.27.5.fc10 set to be updated
--> Processing Dependency: plymouth-plugin-label for package: plymouth-plugin-solar
--> Running transaction check
---> Package plymouth-plugin-label.x86_64 0:0.6.0-0.2008.10.27.5.fc10 set to be updated
--> Finished Dependency Resolution
--> Running transaction check
---> Package kernel.x86_64 0:2.6.26.3-29.fc9 set to be erased
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch     Version                     Repository  Size 
================================================================================
Installing:
 kernel                  x86_64   2.6.27.4-58.fc10            rawhide      21 M
Updating:
 fedora-logos            noarch   10.0.0-2.fc10               rawhide     1.8 M
 initscripts             x86_64   8.84-1                      rawhide     1.9 M
 libbdevid-python        x86_64   6.0.68-1.fc10               rawhide      64 k
 mkinitrd                x86_64   6.0.68-1.fc10               rawhide     112 k
 nash                    x86_64   6.0.68-1.fc10               rawhide     163 k
Removing:
 kernel                  x86_64   2.6.26.3-29.fc9             installed    69 M
Installing for dependencies:
 kernel-firmware         noarch   2.6.27.4-58.fc10            rawhide     345 k
 plymouth                x86_64   0.6.0-0.2008.10.27.5.fc10   rawhide      44 k
 plymouth-libs           x86_64   0.6.0-0.2008.10.27.5.fc10   rawhide      63 k
 plymouth-plugin-label   x86_64   0.6.0-0.2008.10.27.5.fc10   rawhide      15 k
 plymouth-plugin-solar   x86_64   0.6.0-0.2008.10.27.5.fc10   rawhide     455 k
 plymouth-scripts        x86_64   0.6.0-0.2008.10.27.5.fc10   rawhide      13 k

Transaction Summary
================================================================================
Install      7 Package(s)         
Update       5 Package(s)         
Remove       1 Package(s)         

Total download size: 26 M
Is this ok [y/N]: y
Downloading Packages:
(1/12): plymouth-scripts-0.6.0-0.2008.10.27.5.fc10.x86_6 |  13 kB     00:00     
(2/12): plymouth-plugin-label-0.6.0-0.2008.10.27.5.fc10. |  15 kB     00:00     
(3/12): plymouth-0.6.0-0.2008.10.27.5.fc10.x86_64.rpm    |  44 kB     00:00     
(4/12): plymouth-libs-0.6.0-0.2008.10.27.5.fc10.x86_64.r |  63 kB     00:00     
(5/12): libbdevid-python-6.0.68-1.fc10.x86_64.rpm        |  64 kB     00:00     
(6/12): mkinitrd-6.0.68-1.fc10.x86_64.rpm                | 112 kB     00:00     
(7/12): nash-6.0.68-1.fc10.x86_64.rpm                    | 163 kB     00:00     
(8/12): kernel-firmware-2.6.27.4-58.fc10.noarch.rpm      | 345 kB     00:02     
(9/12): plymouth-plugin-solar-0.6.0-0.2008.10.27.5.fc10. | 455 kB     00:03     
(10/12): fedora-logos-10.0.0-2.fc10.noarch.rpm           | 1.8 MB     00:10     
(11/12): initscripts-8.84-1.x86_64.rpm                   | 1.9 MB     00:10     
(12/12): kernel-2.6.27.4-58.fc10.x86_64.rpm              |  21 MB     02:05     
--------------------------------------------------------------------------------
Total                                           169 kB/s |  26 MB     02:35     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : initscripts                                     [ 1/18] 
  Installing     : plymouth-libs                                   [ 2/18] 
  Updating       : nash                                            [ 3/18] 
  Installing     : plymouth-plugin-label                           [ 4/18] 
  Installing     : plymouth-scripts                                [ 5/18] 
  Installing     : plymouth-plugin-solar                           [ 6/18] 
/usr/lib/plymouth/solar.so does not exist
error: %post(plymouth-plugin-solar-0.6.0-0.2008.10.27.5.fc10.x86_64) scriptlet failed, exit status 1
  Installing     : kernel-firmware                                 [ 7/18] 
  Updating       : fedora-logos                                    [ 8/18] 
  Installing     : plymouth                                        [ 9/18] 
  Updating       : mkinitrd                                        [10/18] 
  Installing     : kernel                                          [11/18] 
The default plymouth plugin (.so) doesn't exist
  Updating       : libbdevid-python                                [12/18] 
  Cleanup        : fedora-logos                                    [13/18] 
  Cleanup        : kernel                                          [14/18] 
  Cleanup        : mkinitrd                                        [15/18] 
  Cleanup        : initscripts                                     [16/18] 
  Cleanup        : libbdevid-python                                [17/18] 
  Cleanup        : nash                                            [18/18] 

Removed: kernel.x86_64 0:2.6.26.3-29.fc9
Installed: kernel.x86_64 0:2.6.27.4-58.fc10
Dependency Installed: kernel-firmware.noarch 0:2.6.27.4-58.fc10 plymouth.x86_64 0:0.6.0-0.2008.10.27.5.fc10 plymouth-libs.x86_64 0:0.6.0-0.2008.10.27.5.fc10 plymouth-plugin-label.x86_64 0:0.6.0-0.2008.10.27.5.fc10 plymouth-plugin-solar.x86_64 0:0.6.0-0.2008.10.27.5.fc10 plymouth-scripts.x86_64 0:0.6.0-0.2008.10.27.5.fc10
Updated: fedora-logos.noarch 0:10.0.0-2.fc10 initscripts.x86_64 0:8.84-1 libbdevid-python.x86_64 0:6.0.68-1.fc10 mkinitrd.x86_64 0:6.0.68-1.fc10 nash.x86_64 0:6.0.68-1.fc10
Complete!
[root@xena ~]#

Comment 13 Michael J. Chudobiak 2008-10-29 19:29:33 UTC
Created attachment 321857 [details]
screenshot of /proc and /sys in chrooted rescue

I don't know if this tells you anything useful or not...

Comment 14 Bill Nottingham 2008-10-29 19:34:10 UTC
Sorry, about the continued questions, but /dev is populated in the chroot as well, correct?

Comment 15 Michael J. Chudobiak 2008-10-29 19:47:16 UTC
Yes, /dev is populated in chroot.

Side note: This is on a fairly new HP DC7800 with a newish (ICH9?) chipset. I was trying F10 to get the audio to work. I don't know if that affects anything, but I thought I'd mention it.

- Mike

Comment 16 Bill Nottingham 2008-10-29 20:09:30 UTC
OK. If you do zcat /boot/initrd-xxxx.img | cpio -id "*" (in a temporary dir), what does the /init file look like?

Comment 17 Michael J. Chudobiak 2008-10-30 13:21:25 UTC
Created attachment 321931 [details]
The init file, as requested

Comment 18 Bill Nottingham 2008-11-04 19:14:53 UTC
Does rebuilding the initrd after installing mkinitrd-6.0.69-1 fix it?

Comment 19 Michael J. Chudobiak 2008-11-04 19:41:33 UTC
Yes, rebuilding the initrd after installing mkinitrd-6.0.69-1 fixed it. I have an F10 kernel running now:

[mjc@xena ~]$ uname -a
Linux xena 2.6.27.4-58.fc10.x86_64 #1 SMP Mon Oct 27 17:47:43 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

This bug can be closed, from my vantage point.

Thanks!

- Mike

Comment 20 Bill Nottingham 2008-11-04 19:57:01 UTC
No problem. Glad we could eventually fix it.