Bug 642557 - after a split-media install, system fails to boot
after a split-media install, system fails to boot
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: pungi (Show other bugs)
14
All Linux
low Severity high
: ---
: ---
Assigned To: David Cantrell
Fedora Extras Quality Assurance
:
Depends On:
Blocks: F14Blocker/F14FinalBlocker
  Show dependency treegraph
 
Reported: 2010-10-13 06:47 EDT by Andre Robatino
Modified: 2013-01-10 01:16 EST (History)
13 users (show)

See Also:
Fixed In Version: pungi-2.1.4-1.fc14
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 642617 642621 (view as bug list)
Environment:
Last Closed: 2010-10-19 18:22:11 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs under /tmp after installation before reboot (44.55 KB, application/x-tar)
2010-10-13 07:16 EDT, He Rui
no flags Details
install.log from i386 text install using CDs 1 and 2 (7.96 KB, text/plain)
2010-10-13 07:22 EDT, Andre Robatino
no flags Details
grub.conf from i386 text install using CDs 1 and 2 (797 bytes, text/plain)
2010-10-13 07:23 EDT, Andre Robatino
no flags Details

  None (edit)
Description Andre Robatino 2010-10-13 06:47:04 EDT
Description of problem:
I initially saw this doing a full CD-based i386 install, selecting all packages. I was then able to replicate it with an i386 CD text install (requiring CDs 1 and 2), but not with an x86_64 CD text install (requiring only CD 1). The problem also does not show with an i386 DVD text install. The common thread seems to be whether more than one disc is used during install. I haven't tried a default x86_64 CD install yet.

When attempting to boot after the install, the messages

No root device found

Boot has failed, sleeping forever.

are displayed.

Version-Release number of selected component (if applicable):
F14 Final TC1

How reproducible:
always

Steps to Reproduce:
1. Do a local (non-network) install requiring more than one CD.
2. Reboot afterwards.
  
Actual results:

No root device found

Boot has failed, sleeping forever.

Expected results:

Should boot normally.

Additional info:

At the end of install.log for an i386 DVD text install, I see the messages

Installing plymouth-scripts-0.8.4-0.20100823.4.fc14.i686
Installing plymouth-0.8.4-0.20100823.4.fc14.i686
Need 'inst' function, try setting PLYMOUTH_POPULATE_SOURCE_FUNCTIONS to a file that defines it
Installing dracut-006-2.fc14.noarch
Installing kernel-PAE-2.6.35.6-39.fc14.i686
W: Cannot load dracut module "plymouth", dependencies failed.
*** FINISHED INSTALLING PACKAGES ***

Similar messages appear in x86_64. Could this be related?
Comment 1 Andre Robatino 2010-10-13 06:48:13 EDT
Should have mentioned that I've seen this both in VirtualBox and in KVM. Have not tested on bare metal.
Comment 2 Harald Hoyer 2010-10-13 07:09:09 EDT
(In reply to comment #0)
> Installing plymouth-scripts-0.8.4-0.20100823.4.fc14.i686
> Installing plymouth-0.8.4-0.20100823.4.fc14.i686
> Need 'inst' function, try setting PLYMOUTH_POPULATE_SOURCE_FUNCTIONS to a file
> that defines it
> Installing dracut-006-2.fc14.noarch
> Installing kernel-PAE-2.6.35.6-39.fc14.i686
> W: Cannot load dracut module "plymouth", dependencies failed.
> *** FINISHED INSTALLING PACKAGES ***
> 
> Similar messages appear in x86_64. Could this be related?

Seems like the install of plymouth failed. No dracut is installed yet, and plymouth tries to do in %post:
/usr/libexec/plymouth/plymouth-generate-initrd

which then fails.

kernel does in %posttrans:
/sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update 2.6.35.4-12.fc14.x86_64 

which calls dracut, and dracut cannot find plymouth (because it failed to install in %post)
Comment 3 Harald Hoyer 2010-10-13 07:13:45 EDT
(In reply to comment #0)
> W: Cannot load dracut module "plymouth", dependencies failed.

Although, this indicates, that the "crypt" module can't be loaded (which is a dep of the plymouth module).

The crypt module fails only if "cryptsetup" can't be found. So we have to check, if "cryptsetup-luks" was installed correctly.
Comment 4 He Rui 2010-10-13 07:16:09 EDT
Created attachment 453190 [details]
logs under /tmp after installation before reboot

I can also reproduce it using i386 discs. For the same default text install, x86_64 only requires one disc but i386 needs two: disc#1 and disc#2.
Comment 5 Andre Robatino 2010-10-13 07:20:18 EDT
Should warn that in KVM split-media installs, bug 623126 causes mediacheck to fail on any disc that's bigger than the one originally booted from. Since i386 CD2 is smaller than CD1, that means that mediacheck should be okay when using the first two i386 CDs in a KVM install. VirtualBox isn't affected at all by this issue so it can handle more general tests.
Comment 6 Harald Hoyer 2010-10-13 07:21:20 EDT
any install.log?
Comment 7 Andre Robatino 2010-10-13 07:22:42 EDT
Created attachment 453193 [details]
install.log from i386 text install using CDs 1 and 2
Comment 8 Andre Robatino 2010-10-13 07:23:20 EDT
Created attachment 453194 [details]
grub.conf from i386 text install using CDs 1 and 2
Comment 9 Harald Hoyer 2010-10-13 07:26:06 EDT
no cryptsetup-luks .. hmm.. I might have to put it in the dracut requirements again.
Comment 11 Harald Hoyer 2010-10-13 08:17:25 EDT
(In reply to comment #0)
> When attempting to boot after the install, the messages
> 
> No root device found
> 
> Boot has failed, sleeping forever.

Can you boot with "rdinfo rdshell quiet" on the kernel command line and make a screenshot of:

$ dmesg
Comment 12 Harald Hoyer 2010-10-13 08:17:55 EDT
I mean a photo of the output of "dmesg"
Comment 13 Harald Hoyer 2010-10-13 08:20:22 EDT
Installing dracut-006-2.fc14.noarch
Installing kernel-PAE-2.6.35.6-39.fc14.i686
W: Cannot load dracut module "plymouth", dependencies failed.
Installing lvm2-2.02.73-1.fc14.i686
*** FINISHED INSTALLING PACKAGES ***

Oh!!! lvm2 is installed after dracut has run???

%posttrans is supposed to run after _all_ transactions... weird!
Comment 14 Harald Hoyer 2010-10-13 08:22:51 EDT
(In reply to comment #11)
> (In reply to comment #0)
> > When attempting to boot after the install, the messages
> > 
> > No root device found
> > 
> > Boot has failed, sleeping forever.
> 
> Can you boot with "rdinfo rdshell quiet" on the kernel command line and make a
> screenshot of:
> 
> $ dmesg

scratch that.. I guess there is no lvm in the initramfs...
Comment 15 James Laska 2010-10-13 08:49:06 EDT
(In reply to comment #10)
> This patch is needed for dracut:
> 
> http://dracut.git.sourceforge.net/git/gitweb.cgi?p=dracut/dracut;a=commitdiff;h=5cae1fe179faea815e5adcca71645367829aee01

Filed as bug#642617 - Add cryptsetup-luks to dracut requirements again
Comment 16 James Laska 2010-10-13 08:53:00 EDT
(In reply to comment #2)
> (In reply to comment #0)
> > Installing plymouth-scripts-0.8.4-0.20100823.4.fc14.i686
> > Installing plymouth-0.8.4-0.20100823.4.fc14.i686
> > Need 'inst' function, try setting PLYMOUTH_POPULATE_SOURCE_FUNCTIONS to a file
> > that defines it
> > Installing dracut-006-2.fc14.noarch
> > Installing kernel-PAE-2.6.35.6-39.fc14.i686
> > W: Cannot load dracut module "plymouth", dependencies failed.
> > *** FINISHED INSTALLING PACKAGES ***
> > 
> > Similar messages appear in x86_64. Could this be related?
> 
> Seems like the install of plymouth failed. No dracut is installed yet, and
> plymouth tries to do in %post:
> /usr/libexec/plymouth/plymouth-generate-initrd
> 
> which then fails.
> 
> kernel does in %posttrans:
> /sbin/new-kernel-pkg --package kernel --mkinitrd --dracut --depmod --update
> 2.6.35.4-12.fc14.x86_64 
> 
> which calls dracut, and dracut cannot find plymouth (because it failed to
> install in %post)

Filed as bug#642621 - plymouth %post fails during install
Comment 17 Harald Hoyer 2010-10-13 09:17:33 EDT
so, to summarize:

We see the %posttrans of the kernel rpm before the lvm2 installation.

lvm2 is on CD2, kernel is on CD1.

So, there may be several transactions for each CD and thus %posttrans of CD1 runs before installation of the CD2 packages.

This is very unfortunate in this case, where dracut needs lvm2 to be installed for the initramfs creation.

So, either there is only _one_ %posttrans at the very end, or anaconda needs to run dracut manually afterwards.

I suggest to reassign this bug to component anaconda.
Comment 18 Andre Robatino 2010-10-13 09:43:38 EDT
Did a x86_64 CD install, selecting all packages, which required all 5 CDs. This time the system boots properly afterwards. Let me know if any logs are needed.

https://fedoraproject.org/wiki/QA/TestCases/InstallSourceCdrom
Comment 19 Chris Lumens 2010-10-13 09:48:34 EDT
I suppose the lvm2 package could be added to the following in pungi (bin/pkgorder):

    # hack, hack, hack... make sure iscsi ends up on disc1 (#208832)
    addPackages(ds, ["kernel-*","dracut","mdadm"])
Comment 20 Harald Hoyer 2010-10-13 10:47:31 EDT
(In reply to comment #19)
> I suppose the lvm2 package could be added to the following in pungi
> (bin/pkgorder):
> 
>     # hack, hack, hack... make sure iscsi ends up on disc1 (#208832)
>     addPackages(ds, ["kernel-*","dracut","mdadm"])

add dmraid, cryptsetup-luks, etc., etc... this is error prone..
Comment 21 Bill Nottingham 2010-10-13 11:46:17 EDT
Without introducing hard deps, it seems like the only way to simply fix it.
Comment 22 Harald Hoyer 2010-10-13 11:56:26 EDT
why not run dracut again after everything is installed?
Comment 23 Jesse Keating 2010-10-13 13:24:05 EDT
(In reply to comment #19)
> I suppose the lvm2 package could be added to the following in pungi
> (bin/pkgorder):
> 
>     # hack, hack, hack... make sure iscsi ends up on disc1 (#208832)
>     addPackages(ds, ["kernel-*","dracut","mdadm"])

That's easy enough to do, and certainly not the first time we've had to hack things to get initrd needed things on the first media.
Comment 24 Fedora Update System 2010-10-13 18:53:29 EDT
pungi-2.1.3-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/pungi-2.1.3-1.fc14
Comment 25 Jesse Keating 2010-10-13 20:26:46 EDT
One thing I notice, is that while dracut (and now lvm2) wind up on CD1, things like dracut-network don't.  Is there a scenario where somebody could use anaconda to create an install that would require some of dracut-network to be there for the initrd, and thus fail when doing it from a split media install?
Comment 26 James Laska 2010-10-14 08:19:51 EDT
(In reply to comment #25)
> One thing I notice, is that while dracut (and now lvm2) wind up on CD1, things
> like dracut-network don't.  Is there a scenario where somebody could use
> anaconda to create an install that would require some of dracut-network to be
> there for the initrd, and thus fail when doing it from a split media install?

We haven't tested iSCSI on F-14-Final yet, but based on your comments, I think that may be correct.  If doing an iSCSI root install with split-media, dracut-network wouldn't be installed when the initramfs is created, and we wouldn't be able to boot.
Comment 27 Harald Hoyer 2010-10-14 08:22:34 EDT
(In reply to comment #26)
> (In reply to comment #25)
> > One thing I notice, is that while dracut (and now lvm2) wind up on CD1, things
> > like dracut-network don't.  Is there a scenario where somebody could use
> > anaconda to create an install that would require some of dracut-network to be
> > there for the initrd, and thus fail when doing it from a split media install?
> 
> We haven't tested iSCSI on F-14-Final yet, but based on your comments, I think
> that may be correct.  If doing an iSCSI root install with split-media,
> dracut-network wouldn't be installed when the initramfs is created, and we
> wouldn't be able to boot.

you'll end up with a ton of dependencies and manual crafted pungi rules....

why not run dracut at the end like you do for "grub-install" ??
Comment 28 Harald Hoyer 2010-10-14 08:27:54 EDT
    addPackages(ds, ["kernel-*","dracut","mdadm",
                     "lvm2", "device-mapper", "plymouth", 
                     "cryptsetup-luks", "dmraid", "kbd", 
                     "plymouth-scripts", "plymouth", "dracut-network" ])
Comment 29 Harald Hoyer 2010-10-14 08:29:22 EDT
    addPackages(ds, ["kernel-*","dracut","mdadm",
                     "lvm2", "device-mapper", "plymouth", 
                     "cryptsetup-luks", "dmraid", "kbd", 
                     "plymouth-scripts", "plymouth", "dracut-network",
                     "multipath" ])
Comment 30 James Laska 2010-10-14 08:43:10 EDT
(In reply to comment #27)
> why not run dracut at the end like you do for "grub-install" ??

Can anyone from the anaconda-devel side (cc'd) speak to this?
Comment 31 Peter Jones 2010-10-14 09:06:14 EDT
dracut _is_ run at the end - it's run by new-kernel-pkg in kernel's %posttrans .
Comment 32 Peter Jones 2010-10-14 09:17:42 EDT
That is, it's run at the end of the kernel's transaction - which on split media will be the end of disc 1.
Comment 33 Fedora Update System 2010-10-14 09:52:34 EDT
pungi-2.1.3-1.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update pungi'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/pungi-2.1.3-1.fc14
Comment 34 David Lehman 2010-10-14 11:23:09 EDT
(In reply to comment #30)
> (In reply to comment #27)
> > why not run dracut at the end like you do for "grub-install" ??
> 
> Can anyone from the anaconda-devel side (cc'd) speak to this?

The problem is not in anaconda. It's either in the tools that make the media or in rpm, which suggests that cramming a hack into anaconda to fix it makes less than total sense. Anaconda does not want to be a collection of hacks to cover bugs in other packages.
Comment 35 Jesse Keating 2010-10-14 13:58:08 EDT
I've amended the latest pungi build with the above list of initrd related packages.  New bodhi ticket coming soon.
Comment 36 Fedora Update System 2010-10-14 14:02:11 EDT
pungi-2.1.4-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/pungi-2.1.4-1.fc14
Comment 37 Fedora Update System 2010-10-19 17:25:43 EDT
spin-kickstarts-0.14.4-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/spin-kickstarts-0.14.4-1.fc14
Comment 38 Fedora Update System 2010-10-19 18:22:05 EDT
spin-kickstarts-0.14.4-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.
Comment 39 Fedora Update System 2010-10-19 18:24:57 EDT
pungi-2.1.4-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.