Bug 970580

Summary: FedUp 18->19 Fails to start RAID5 during first reboot
Product: [Fedora] Fedora Reporter: Dave Ludlow <dave>
Component: fedupAssignee: Will Woods <wwoods>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 19CC: appublic+bugzillaredhat, bill-bugzilla.redhat.com, chewi, collura, gholms, mr.nuke.me, redhat-bugzilla, tflink, udovdh, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-08-16 18:40:52 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Ludlow 2013-06-04 12:03:02 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:21.0) Gecko/20100101 Firefox/21.0
Build Identifier: 

On initial reboot after running "fedup --network 19" I was dropped to a dracut emergency shell with errors relating to being unable to find /dev/md0.  Running "ls -l /dev/md0" did not show any such file.  I ran "madam --assemble", and that started my raid and created the file.  I typed "exit" to quit the emergency shell and the upgrade proceeded from there.  Although I was unable to see the GUI (the screen was stuck in text mode with my previous dracut prompt), the upgrade completed successfully.

This system has a root on RAID 5.  I don't have any way to go back and re-run the upgrade for diagnostic purposes.


Reproducible: Didn't try

Steps to Reproduce:
1. "fedup --network 19" on a Fedora 18 x86_64 system with a RAID 5 root

Actual Results:  
Dracut emergency shell with no /dev/md0

Expected Results:  
Proceed to upgrade GUI

I was able to work around the problem by running "mdadm --assemble" and then exiting the dracut emergency shell.

Comment 1 Andrew 2013-07-02 22:20:11 UTC
I experienced the identical issue and resolution, although my /dev/md1 is a RAID 1 mirror.  The MDs were clearly not assembled during the fedup boot process.

Comment 2 Will Woods 2013-07-03 22:13:10 UTC
I'm guessing this is probably the same as bug 974000, but for rd.md.* instead of rd.luks.*.

1) What are the boot arguments from the F18 kernel?
2) What boot arguments were used with fedup?
3) Did you migrate from GRUB to GRUB2 by hand? If so, are there any boot arguments from the old configuration that you didn't put into the new config?

Comment 3 Will Woods 2013-07-08 16:41:42 UTC
*** Bug 981824 has been marked as a duplicate of this bug. ***

Comment 4 Bill McGonigle 2013-07-09 05:06:47 UTC
Seeing this on a machine with both LUKS and md.  I get much further than I was, with rd.auto=1, but after fedup has verified the RPM transaction, I get:

starting upgrade...
error: can't create transaction lock on /sysroot/var/lib/rpm/.rpm.lock (read-only filesystem)
upgrade finished with non-fatal errors <----- DID NOT! :)
upgrade finished

messages:
 https://www.youtube.com/watch?v=SB_SmL_A0ZQ

to provide the requested data, but for my machine:

1) linux   /vmlinuz-3.9.6-200.fc18.x86_64 root=UUID=94f30de7-513e-4cd4-ba08-e76db67166f3 ro enforcing=0 elevator=deadline
 rd.auto=1
2) hrm, this time around it's writing a grub.conf, not a grub2 entry (not sure why).  Anyway, that looks like:

kernel /vmlinuz-3.9.6-200.fc18.x86_64 ro root=/dev/mapper/luks-278acbeb-1edf-4d92-9570-19c95c88e147 rd_LUKS_UUID=luks-278acbeb-1edf-4d92-9570-19c95c88e147 rd_MD_UUID=46aa563e:a3f3b169:17b6030e:a566c47d rd_LUKS_UUID=luks-76c80885-bbf9-4e0d-8909-041c10fc98cd rd_MD_UUID=df05c5cb:51b55202:999785f0:e2339a82 rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us radeon.audio=0 plymouth.debug=file:/run/plymouth/debug.log enforcing=0 upgrade systemd.unit=system-upgrade.target plymouth.splash=fedup selinux=0 rd.auto=1

but grub2 is install the bootloader so it's not in my boot menu.

3) just upgraded to grub2 this morning from a f17-f18 update (an ugly upgrade with yum, didn't know why fedup was failing on boot md0 at the time).  kernel upgrades have erased all the old entries.

Comment 5 Bill McGonigle 2013-07-09 12:38:14 UTC
followup: I ran fedup again and got a grub2 entry:

linux   /vmlinuz-fedup root=/dev/mapper/luks-278acbeb-1edf-4d92-9570-19c95c88e147 ro enforcing=0 elevator=deadline rd.auto=1 upgrade systemd.unit=system-upgrade.target plymouth.splash=fedup selinux=0

Comment 6 Michael Chapman 2013-07-09 14:19:13 UTC
For what it's worth, I had this same problem with root-on-LVM-on-RAID1.

I don't normally use any rd.* command-line parameters -- I prefer to just let dracut autodetect everything it needs. I assume fedup won't have added any rd.* parameters itself. (I can't be sure now that I've completed the upgrade... and unfortunately this info wasn't logged in fedup.log.)

Assuming it would have copied most of the details from my previous F18 kernel command-line, it would have just contained:

root=/dev/beren/root ro rhgb quiet vconsole.font=latarcyrheb-sun16 locale.LANG=en_AU.UTF-8 vconsole.keymap=us

Comment 7 Bill McGonigle 2013-07-09 15:57:36 UTC
FYI, here's my blkid.  I tried commenting everything but / (ext4 on luks on md1;sda2+sdb2/raid1) and /boot (ext4 on md0;sda1+sdb1/raid1) from crypttab and fstab before running fedup and still didn't succeed with the upgrade.  Swap is typically on luks on md2;sda3+sdb3/raid1.  /home is handled by zfs on sda4/sdb4 with caches on sdc partitions (removing /home from the equation didn't change the outcome).

/dev/sda1: UUID="8b642553-8f54-2e6c-05bd-6a3518f14b63" UUID_SUB="59f48a8e-c2b2-1a9c-723c-11dd63617ea1" LABEL="localhost.localdomain:0" TYPE="linux_raid_member" 
/dev/sda2: UUID="46aa563e-a3f3-b169-17b6-030ea566c47d" UUID_SUB="105f574a-0ae1-0b90-f97a-9f9d02e29e7f" LABEL="localhost.localdomain:1" TYPE="linux_raid_member" 
/dev/sda3: UUID="df05c5cb-51b5-5202-9997-85f0e2339a82" UUID_SUB="ebb610fd-e470-fe93-26d9-886c3a2556f5" LABEL="localhost.localdomain:2" TYPE="linux_raid_member" 
/dev/sda4: UUID="58d5a9b4-6ca0-4960-b7bd-c2a4f72e6e6e" TYPE="crypto_LUKS" 
/dev/sdb1: UUID="8b642553-8f54-2e6c-05bd-6a3518f14b63" UUID_SUB="b67d64e6-eae9-c09c-fe19-5d0b2428ca3a" LABEL="localhost.localdomain:0" TYPE="linux_raid_member" 
/dev/sdb2: UUID="46aa563e-a3f3-b169-17b6-030ea566c47d" UUID_SUB="8ff06c7c-6da0-6109-a2dd-02b11fde0a31" LABEL="localhost.localdomain:1" TYPE="linux_raid_member" 
/dev/sdb3: UUID="df05c5cb-51b5-5202-9997-85f0e2339a82" UUID_SUB="08d76549-a652-c5c7-7372-3f386db035a2" LABEL="localhost.localdomain:2" TYPE="linux_raid_member" 
/dev/sdb4: UUID="501f1d41-6d24-48e4-a7b4-c5b9b8b12a58" TYPE="crypto_LUKS" 
/dev/sdc1: UUID="004bef84-4a1f-4a2b-b2a8-4807333a0bff" TYPE="crypto_LUKS" PARTUUID="d9a480bb-7fbd-4afe-8688-2310cd3561b0" 
/dev/sdc4: UUID="5f81bd5d-8546-42d0-977b-95935f9f0ef1" TYPE="crypto_LUKS" PARTUUID="f75e71a4-6ab1-4c9d-a639-5079e05a154a" 
/dev/md0: UUID="2e0b053a-a3b2-46e9-9507-fd85f082574d" TYPE="ext4" 
/dev/md2: UUID="76c80885-bbf9-4e0d-8909-041c10fc98cd" TYPE="crypto_LUKS" 
/dev/md1: UUID="278acbeb-1edf-4d92-9570-19c95c88e147" TYPE="crypto_LUKS" 
/dev/mapper/luks-278acbeb-1edf-4d92-9570-19c95c88e147: LABEL="_Fedora-14-x86_6" UUID="94f30de7-513e-4cd4-ba08-e76db67166f3" TYPE="ext4" 
/dev/mapper/luks-5f81bd5d-8546-42d0-977b-95935f9f0ef1: LABEL="home" UUID="11586581848017399922" UUID_SUB="7380371109528257841" TYPE="zfs_member" 
/dev/mapper/luks-76c80885-bbf9-4e0d-8909-041c10fc98cd: UUID="16205cf3-7f94-42db-bde5-b887d42a16b9" TYPE="swap" 
/dev/mapper/luks-501f1d41-6d24-48e4-a7b4-c5b9b8b12a58: LABEL="home" UUID="11586581848017399922" UUID_SUB="875733894015943285" TYPE="zfs_member" 
/dev/mapper/luks-58d5a9b4-6ca0-4960-b7bd-c2a4f72e6e6e: LABEL="home" UUID="11586581848017399922" UUID_SUB="15369474955789209120" TYPE="zfs_member" 
/dev/sdc2: PARTUUID="b688fe46-8146-48d0-bd6a-e6483b93a35e" 
/dev/sdc3: PARTUUID="08ce8573-4172-4b03-ae19-ed24fee214c5" 
/dev/sdc5: PARTUUID="c4878442-2044-4455-99f0-3f1cfb637b09" 
/dev/sdc6: PARTUUID="8039e124-dde8-4a41-a957-663331e2e2af"

Comment 8 Dave Ludlow 2013-07-09 20:31:26 UTC
1) What are the boot arguments from the F18 kernel?
  Don't know any more
2) What boot arguments were used with fedup?
  fedup --network 19
3) Did you migrate from GRUB to GRUB2 by hand? If so, are there any boot arguments from the old configuration that you didn't put into the new config?
  Probably, years ago, so I don't know.  Sorry!

Comment 9 James Le Cuirot 2013-07-12 22:08:20 UTC
It seems that rd.auto=1 fixes it for me too, though I don't have local access to the machine until Monday so I only tried an initial dummy run over SSH using QEMU in snapshot mode. My setup is LVM on LUKS on RAID.

Comment 10 James Le Cuirot 2013-07-12 22:13:03 UTC
1) What are the boot arguments from the F18 kernel?
  root=/dev/mapper/vg_red-lv_root ro quiet rhgb LANG=en_GB.UTF-8
2) What boot arguments were used with fedup?
  fedup --network 19
3) Did you migrate from GRUB to GRUB2 by hand? If so, are there any boot arguments from the old configuration that you didn't put into the new config?
  Can't remember either, sorry! :|

Before trying rd.auto=1, I did try adding rd.luks.uuid first but I guess that wasn't sufficient because I needed the md stuff too.

Comment 11 Will Woods 2013-07-16 16:15:32 UTC
*** Bug 984213 has been marked as a duplicate of this bug. ***

Comment 12 Will Woods 2013-08-16 18:40:52 UTC
Yep, the common problem here is that you're missing the (necessary as of F19) rd.{md,lvm,luks}.* arguments, which makes this a duplicate of 974000.

If you encounter the problem, you should:

1) run "fedup --reset-bootloader"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This just removes the fedup boot entry.


2) edit your GRUB config and add 'rd.md.uuid=XXX' for your RAID device
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You might also need rd.luks.uuid=XXX and/or rd.lvm.lv=XXX, depending on your system setup; your old grub.cfg should have had these values written by the installer.

If you're upgrading from F18 you can use 'rd.auto=0' to simulate the F19 behavior and make sure you have your boot arguments right.

If you can't find the correct arguments, 'rd.auto=1' will tell dracut to just auto-assemble everything like it did in F18 and earlier. 

Once you're sure you have boot arguments that will work in F19, you can move on to:

3) re-run fedup with the same arguments you did before
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It will keep all previously-downloaded packages/images and only download new updates (if any), and then it will set up the fedup boot entry using the arguments you set up in step 2, and then your upgrade should work.


NOTE: do *not* just add 'rd.auto=1' (or whatever) to the fedup boot entry! The fedup boot entry will be thrown away when fedup starts, so the upgrade will work but the resulting F19 boot entry won't have the right boot arguments and your system won't boot.

*** This bug has been marked as a duplicate of bug 974000 ***

Comment 13 udo 2013-08-17 04:39:39 UTC
Please explain why my kernel.org, compiled in fedora 19, boots without all this stuff.
Please explain why fedup cannot prepare this.
Please explain why the installer is not on the iso anymore.