Bug 895805 - Software RAID root /dev/md1 assembled as /dev/md127 during fedup-upgrade boot, root mount fails
Summary: Software RAID root /dev/md1 assembled as /dev/md127 during fedup-upgrade boot...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: fedup-dracut
Version: 18
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
Assignee: Will Woods
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-16 02:55 UTC by Andrew
Modified: 2013-07-08 14:25 UTC (History)
10 users (show)

Fixed In Version: fedup-0.7.3-5.fc17
Clone Of:
Environment:
Last Closed: 2013-05-15 03:30:43 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
disc configuration after fedup execution before reboot (9.13 KB, text/plain)
2013-05-07 15:29 UTC, Tony
no flags Details
disc configuration after fail upgrade reboot in emergency mode (2.51 KB, text/plain)
2013-05-07 15:31 UTC, Tony
no flags Details

Description Andrew 2013-01-16 02:55:19 UTC
Description of problem:

MY Fedora 17 system runs two software raid volumes:

/boot = /dev/md0
/ = /dev/md1

Normal boot routine has kernel parameter of root=/dev/md1.

I ran fedup-cli to upgrade to Fedora 18.  The pre-reboot process ran normally.  On reboot, the RAID is assembled as /dev/md127 instead of /dev/md1, so the filesystem mounting process fails.


Version-Release number of selected component (if applicable):

Not sure.

How reproducible:

Run fedup-cli on a Fedora 17 system with a /dev/md0 or /dev/md1 RAID and the fedup-upgrade kernel will assemble as something else, e.g., /dev/md126

Steps to Reproduce:
1.
2.
3.
  
Actual results:

/dev/md0 is assembled as /dev/md126
/dev/md1 is assembled as /dev/md127

Expected results:

/dev/md0 assembled as /dev/md0
/dev/md1 assembled as /dev/md1


Additional info:

Comment 1 Daniel 2013-01-16 13:01:53 UTC
I've encountered the same issue on a system with two software raid volumes 
/boot = /dev/md0
/ = /dev/md1

Booting to fedup-upgrade kernel, I have md126 and md127.
I tried to change kernel parameters to root=/dev/md127, but the system hangs or prints unlimited '*' chars.

Comment 2 Michael 2013-01-18 17:56:12 UTC
The same happen with me.

I run the Fedup to my PC with Fedora 17 and after the reboot my 2 Raid volumes (md0 =/boot & md1=/) became md 126 & md127.

I tried to rename to md0 & md1 with mdadm --stop /dev/md* and 
mdadm --assemble --scan 

but i have problem with the md127 (the old md1 = /) 

mdadm --stop /dev/md127
mdadm: Cannot get exclusive access to /dev/md127:Perhaps a running process, mounted filesystem or active volume group?

Comment 3 Michael 2013-01-22 09:22:53 UTC
I check the : /boot/initramfs-fedup.img and DOES NOT contain the mdadm.conf !

Maybe this is the reason for the problem with RAID.....

How i can update this file to include the mdadm.conf ??

Comment 4 Michael 2013-01-23 15:30:36 UTC
The solution i found and works for me is to include my mdstat.conf in the initramfs-fedup.img

1. First uncompress the initramfs-fedup.img to a folder using ARK utility.
This create several folders and files like /etc, /var, /run, etc.

2. Copy from /etc/mdstat.conf to the /etc/ folder.

3. Re-compress the folders and create a new initramfs-fedup, using :

Code:
find . | cpio -H newc -o | xz --check=crc32 --x86 --lzma2 > /initramfs-fedup.img

I use the xz and not gzip compress, as the original image is xz compressed.
This create a identical image of fedup including my mdstat.conf at /etc/

4. Replace the original \boot\initramfs-fedup.img with this i create.

5. Reboot and Fedup recognize my RAID Disks and continue upgrade procedure.


For more information of unpack and repack initramfs.img look :

http://www.sysresccd.org/forums/viewtopic.php?f=25&t=4226
http://linux.koolsolutions.com/2009/...pressed-image/

Comment 5 Will Woods 2013-01-23 16:15:02 UTC
Why would your system care whether it's /dev/md0 or /dev/md126? If your /etc/fstab is using UUID=XXX or LABEL=... it shouldn't matter, as far as I know.

Could you please attach your /etc/fstab?

And: if there are hardcoded references to /dev/mdX, try changing them to UUID=XXX instead (use `blkid` to find the UUIDs).

Comment 6 Michael 2013-01-23 17:57:36 UTC
#
# /etc/fstab
# Created by anaconda on Fri Nov 14 19:37:17 2008
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or vol_id(8) for more info
#
/dev/VolGroup10/LogVol00 /                       ext3    defaults        1 1
/dev/VolGroup10/LogVol02		/home		ext4	defaults	1 2
/dev/md0                /boot                   ext3    defaults        1 2
/dev/VolGroup10/LogVol01 /var                    ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
UUID=a6efc786-895a-4744-a6b9-1ac5079c8676 swap                    swap    defaults        0 0
UUID=99862119-097a-42bf-bca9-ee42f46699c7 swap                    swap    defaults        0 0
/dev/sda1		/home/RAID5		ext4	defaults	1 2
/dev/sdd1		/home/HDD2TB		ext4	defaults	1 2

Comment 7 Andrew 2013-01-23 18:14:46 UTC
Fair enough...my /etc/fstab does refer to /dev/mdX (this is legacy, I assume, since I have never mucked with this file).  However, it seems like incorporating the mdadm.conf or mdstat into the initramfs would make sense, so that the upgrade will work for any legal /etc/fstab reference.  Making people muck with their /etc/fstab to get it to work seems risky and unnecessary.

Comment 8 Michael 2013-01-23 18:18:15 UTC
As i saw when the Fedup reboot, recognize the md1 and attach all LogVol0* but not recognize the md0 (/boot) and give me the following error :

[DEPEND] Dependency failed for dev-md0.device 

and exit me to emergency console.

Comment 9 Will Woods 2013-01-23 20:43:37 UTC
Using /dev/mdX in /etc/fstab is known to be risky. Which is why we don't do that anymore.

It might be possible to pull mdadm.conf into the initramfs to fix this, but right now I need to know whether or not replacing the '/dev/mdX' references with 'UUID=XXX' in /etc/fstab *also* fixes the problem.

Comment 10 Andrew 2013-01-23 20:47:38 UTC
Thanks, Will; makes sense.  Unfortunately, I can't test this for you...I had done upgrades (as opposed to fresh installs) since ~ Fedora 12 (which is probably why I still had the /dev/mdX format in my /etc/fstab).  Once I ran into this issue, I used it as impetus to do a fresh install with Fedora 18 on my system (and /etc/fstab does now use UUID=).  Hopefully one of the other folks can test this for you.

Comment 11 Michael 2013-01-23 21:39:23 UTC
I've already upgraded to Fedora 18, using the method I described in my previous message. So i can't test it until next upgrade.

Comment 12 Michael 2013-01-27 20:19:34 UTC
Please look at : http://forums.fedoraforum.org/showthread.php?p=1629531#post1629531   message #14 it might help you.

Comment 13 Fedora Update System 2013-03-15 23:35:24 UTC
fedup-0.7.3-1.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-1.fc17

Comment 14 Fedora Update System 2013-03-17 00:55:32 UTC
Package fedup-0.7.3-1.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing fedup-0.7.3-1.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-3956/fedup-0.7.3-1.fc17
then log in and leave karma (feedback).

Comment 15 Will Woods 2013-04-22 16:04:35 UTC
Can anyone confirm that fedup-0.7.3-1.fc17 fixes the problem for mdraid systems using mdadm.conf?

Comment 16 Eric Princen 2013-04-25 06:14:43 UTC
(In reply to comment #15)
> Can anyone confirm that fedup-0.7.3-1.fc17 fixes the problem for mdraid
> systems using mdadm.conf?

I just tried it. Didn't work. My fedup fails to update with raid as well. Dependency problem starting up trying to mount /home. I'm assuming its not mounting /. I'm on my iPad now, so I don't have the boot failure report handy. Preupgrade had handled this configuration properly in the past.

Comment 17 Eric Princen 2013-04-25 06:20:39 UTC
Btw, this is my second dev box. My primary had the same problem, so I decided to finally check out Ubuntu, and I like it. I may not leave the F17 box in this state for much longer, so if you need info and testing, it's going to have to be soon. Just an FYI so you can take advantage of my testbed while it stays a Fedora box, and if the upgrade takes, I can avoid dealing with copying my /home twice. :-)

Comment 18 Fedora Update System 2013-04-30 14:52:21 UTC
fedup-0.7.3-3.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc18

Comment 19 Fedora Update System 2013-04-30 14:56:18 UTC
fedup-0.7.3-3.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-3.fc17

Comment 20 Fedora Update System 2013-04-30 15:38:39 UTC
fedup-0.7.3-4.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc17

Comment 21 Fedora Update System 2013-04-30 15:39:20 UTC
fedup-0.7.3-4.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc18

Comment 22 Fedora Update System 2013-04-30 15:40:14 UTC
fedup-0.7.3-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/fedup-0.7.3-4.fc19

Comment 23 Tony 2013-05-07 07:42:30 UTC
Tried fedup on fedora 17 x64 with multiple RAID arrays.

after initramfs-fedup.img was downloaded get :

# xz -tv /boot/initramfs-fedup.img
/boot/initramfs-fedup.img (1/1)
  100 %         26,0 MiB / 88,9 MiB = 0,293                   0:02


but after fedup finish get :

# xz -tv /boot/initramfs-fedup.img
/boot/initramfs-fedup.img (1/1)
 99,9 %         26,0 MiB / 88,9 MiB = 0,293                   0:01             
xz: /boot/initramfs-fedup.img: Compressed data is corrupt
 99,9 %         26,0 MiB / 88,9 MiB = 0,293


and no error when appending things to initrd in log file:

# tail fedupdebug.log 
[  1969.129] (II) fedup.rpm:pipelogger() got EOF
[  1969.141] (II) fedup.sysprep:link_pkgs() linking required packages into packagedir
[  1969.141] (II) fedup.sysprep:link_pkgs() packagedir = /var/lib/fedora-upgrade
[  1969.142] (II) fedup.rpm:pipelogger() exiting
[  1969.270] (II) fedup.sysprep:setup_upgradelink() setting up upgrade symlink: /system-upgrade->/var/lib/fedora-upgrade
[  1969.270] (II) fedup.sysprep:setup_upgraderoot() creating upgraderoot dir: /system-upgrade-root
[  1969.271] (II) fedup.sysprep:prep_boot() appending /etc/mdadm.conf to initrd
[  1969.330] (II) fedup.sysprep:prep_boot() found updates in /etc/fedup/update.img.d, appending to initrd
[  1969.344] (II) fedup.sysprep:modify_bootloader() adding new boot entry
[  1969.566] (II) fedup:<module>() /bin/fedup exiting at Tue May  7 09:07:58 2013

Comment 24 Tony 2013-05-07 15:29:55 UTC
Created attachment 744789 [details]
disc configuration after fedup execution before reboot

Comment 25 Tony 2013-05-07 15:31:59 UTC
Created attachment 744790 [details]
disc configuration after fail upgrade reboot in emergency mode

After reboot to 'System Upgrade (fedup)' failing to Emergency complaining about /var array timed out and only root array mounted.

Does it's going like https://bugzilla.redhat.com/show_bug.cgi?id=748119 ?

Comment 26 Fedora Update System 2013-05-15 03:30:43 UTC
fedup-0.7.3-4.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 27 marodoc 2013-06-03 11:17:03 UTC
Can this thread be resurrected?
This issue still seems to exist with 'fedup.noarch 0.7.3-4.fc17'.

First I got timeout message in '/var/log/fedup.log' saying that the operation to download some packages was too slow. I reran 'sudo fedup-cli --network 18' and these packages were correctly downloaded.

I rebooted and saw message that /raid couldn't be mounted (can't get hold of the full message now).

It looks like I'm running dmraid system and I'm using mdadm.conf.

Since I tried it 2 times with the same result I went ahead and executed upgrade via yum.

Some problems with grub2 (error: file '/grub2/locale/en.mo.gz' not found) resolved by https://bugzilla.redhat.com/show_bug.cgi?id=817187#c25

Now still during booting, screen shows old-school three-coloured progress bar at the bottom of the screen (just before login screen). After logging in it takes good few seconds for the OS to be responsive.

I guess now it's too late for me to help to resolve this bug but I'm happy to provide any logs if they still exist.

Comment 28 Fedora Update System 2013-06-07 16:44:03 UTC
fedup-0.7.3-5.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/fedup-0.7.3-5.fc17

Comment 29 Fedora Update System 2013-06-08 03:35:17 UTC
fedup-0.7.3-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 30 Fedora Update System 2013-06-23 05:57:18 UTC
fedup-0.7.3-5.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.