Bug 732133 - Not booting from MDRAID /boot
Not booting from MDRAID /boot
Status: CLOSED DUPLICATE of bug 732164
Product: Fedora
Classification: Fedora
Component: grub2 (Show other bugs)
16
i686 Unspecified
unspecified Severity medium
: ---
: ---
Assigned To: Peter Jones
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-08-19 18:04 EDT by Tom H
Modified: 2011-09-23 21:40 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-09-23 21:40:43 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Tom H 2011-08-19 18:04:51 EDT
Description of problem:
The installation of F16 on an MDRAID array succeeds but rebooting into the new install drops you to a "grub rescue" prompt.


Version-Release number of selected component (if applicable):
grub2
version 1.99
release 0.2.fc16


How reproducible:
Install F16 on an MDRAID array.


Steps to Reproduce:
1. Install F16 on an MDRAID array.
2. Reboot.
3.


Actual results:
Boot to "grub rescue" prompt.

Expected results:
Boot to login prompt.
Comment 1 Tom H 2011-08-19 18:05:38 EDT
[root@localhost ~]# parted -l
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system     Name           Flags
 1      1049kB  2097kB  1049kB                                 bios_grub
 2      2097kB  8181MB  8179MB  ext4            software RAID  boot
 3      8181MB  8589MB  408MB   linux-swap(v1)

Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sdb: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system     Name           Flags
 1      1049kB  2097kB  1049kB                                 bios_grub
 2      2097kB  8181MB  8179MB  ext4            software RAID  boot
 3      8181MB  8589MB  408MB   linux-swap(v1)

Model: Linux Software RAID Array (md)
Disk /dev/md0: 8179MB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number  Start  End     Size    File system  Flags
 1      0.00B  8179MB  8179MB  ext4
Comment 2 Tom H 2011-08-19 18:07:48 EDT
At the "grub rescue" prompt "set" outputs:
prefix=(hd0)/boot/grub2
root=(hd0)

At the "grub rescue" prompt "ls" outputs:
(hd0) (hd1)
Comment 3 Tom H 2011-08-19 18:09:38 EDT
/var/log/anaconda/anaconda.program.log ends with:
19:19:40,876 INFO program: Running... grub2-install (hd0)
19:19:52,095 INFO program: Installation finished. No error reported.
19:19:52,098 INFO program: Running... grub2-install (hd1)
19:20:03,112 INFO program: Installation finished. No error reported.
Comment 4 Tom H 2011-08-19 18:34:52 EDT
Booted from the installation DVD into its rescue mode and checked grub.cfg (see below). It had "root=(md0)" and the correct UUID entries but no gpt or raid insmod lines.

I added "insmod part_gpt", "insmod raid", insmod "mdraid1x", rebooted, and ended up at the "grub rescue prompt" again.

[root@localhost ~]# cat /etc/grub2.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  insmod vbe
  insmod vga
  insmod video_bochs
  insmod video_cirrus
}

set timeout=20
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Fedora Linux, with Linux 3.0.0-1.fc16.i686.PAE' --class fedora --class gnu-linux --class gnu --class os {
	load_video
	set gfxpayload=keep
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set=root d926a2c0-3c34-413b-966d-7281ac8d3f7a
	echo	'Loading Linux 3.0.0-1.fc16.i686.PAE ...'
	linux	/boot/vmlinuz-3.0.0-1.fc16.i686.PAE root=UUID=d926a2c0-3c34-413b-966d-7281ac8d3f7a ro rd.lvm=0 rd.dm=0 rd.md.uuid=9f679bf8:078e4255:3be4632f:f10ee451  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8 
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initramfs-3.0.0-1.fc16.i686.PAE.img
}
menuentry 'Fedora Linux, with Linux 3.0.0-1.fc16.i686.PAE (recovery mode)' --class fedora --class gnu-linux --class gnu --class os {
	load_video
	set gfxpayload=keep
	insmod ext2
	set root='(md0)'
	search --no-floppy --fs-uuid --set=root d926a2c0-3c34-413b-966d-7281ac8d3f7a
	echo	'Loading Linux 3.0.0-1.fc16.i686.PAE ...'
	linux	/boot/vmlinuz-3.0.0-1.fc16.i686.PAE root=UUID=d926a2c0-3c34-413b-966d-7281ac8d3f7a ro single rd.lvm=0 rd.dm=0 rd.md.uuid=9f679bf8:078e4255:3be4632f:f10ee451  KEYTABLE=us quiet SYSFONT=latarcyrheb-sun16 rhgb rd.luks=0 LANG=en_US.UTF-8
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initramfs-3.0.0-1.fc16.i686.PAE.img
}
### END /etc/grub.d/10_linux ###
<other (empty) sections snipped>
Comment 5 Tom H 2011-08-19 18:43:41 EDT
Booted from the installation DVD into its rescue mode and re-installed grub2 with grub2-setup:

grub2-setup --verbose --core-image=core2.img --directory=/boot/grub2 --device-map=/boot/grub2/device.map /dev/sda
grub2-setup --verbose --core-image=core2.img --directory=/boot/grub2 --device-map=/boot/grub2/device.map /dev/sdb

with core2.img built with:

grub2-mkimage --verbose --directory=/usr/lib/grub2/i386-pc --prefix=/boot/grub2 --output=/boot/grub2/core2.img --format=i386-pc biosdisk boot configfile ext2 extcmd fshelp gzio mdraid1x minicmd normal part_gpt raid terminal

(I wasn't sure which modules are added by default so I checked with "lsmod" on another install which modules were loaded and added the raid ones)

where

[root@localhost ~]# cat /boot/grub2/device.map
# this device map was generated by anaconda
(hd0)      /dev/sda
(hd1)      /dev/sdb
(md0)      /dev/md0

Rebooted to a "grub" prompt.

"set" outputs:
prefix=(hd0)/boot/grub2
root=(hd0)

"ls" outputs (not in this order but I don;t have that screen in front of me):
(hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3) (hd1) (hd1,gpt1) (hd1,gpt2) (hd1,gpt3)

No "(md0)" in spite of "lsmod" showing that "raid" and "mdraid1x" are loaded.

But I can now boot with:
set prefix=(hd0,gpt2)/boot/grub2
set root=(hd0,gpt2)
linux /boot/vmlinuz... root=/dev/md0 ro
initrd /boot/initramfs...
boot
Comment 6 Tom H 2011-08-19 18:45:59 EDT
grub2-probe outputs everything correctly except for abstraction:

[root@localhost ~]# grub2-probe --target=fs --device /dev/sda2
ext2
[root@localhost ~]# grub2-probe --target=fs --device /dev/sdb2
ext2
[root@localhost ~]# grub2-probe --target=fs --device /dev/md0
ext2
[root@localhost ~]# grub2-probe --target=fs /
ext2
[root@localhost ~]# grub2-probe --target=fs_uuid --device /dev/sda2
d926a2c0-3c34-413b-966d-7281ac8d3f7a
[root@localhost ~]# grub2-probe --target=fs_uuid --device /dev/sdb2
d926a2c0-3c34-413b-966d-7281ac8d3f7a
[root@localhost ~]# grub2-probe --target=fs_uuid --device /dev/md0
d926a2c0-3c34-413b-966d-7281ac8d3f7a
[root@localhost ~]# grub2-probe --target=fs_uuid /
d926a2c0-3c34-413b-966d-7281ac8d3f7a
[root@localhost ~]# grub2-probe --target=drive --device /dev/sda
(hd0)
[root@localhost ~]# grub2-probe --target=drive --device /dev/sda2
(hd0,gpt2)
[root@localhost ~]# grub2-probe --target=drive --device /dev/sdb
(hd1)
[root@localhost ~]# grub2-probe --target=drive --device /dev/sdb2
(hd1,gpt2)
[root@localhost ~]# grub2-probe --target=drive --device /dev/md0
(md0)
[root@localhost ~]# grub2-probe --target=device /
/dev/md0
[root@localhost ~]# grub2-probe --target=partmap --device /dev/sda2
gpt
[root@localhost ~]# grub2-probe --target=partmap --device /dev/sdb2
gpt
[root@localhost ~]# grub2-probe --target=abstraction --device /dev/sda2

[root@localhost ~]# grub2-probe --target=abstraction --device /dev/sdb2

[root@localhost ~]# grub2-probe --target=abstraction --device /dev/md0

[root@localhost ~]# grub2-probe --target=abstraction /

[root@localhost ~]#
Comment 7 Bruno Wolff III 2011-08-19 19:57:52 EDT
I am not sure if things were planned to change for grub2 in F16, but normally grub treats raid 1 array elements as individual file systems and accesses them read only. For this to work the arrays need to be version 0.90 or 1.0. 1.1 and 1.2 arrays will prevent that trick from working.
Comment 8 Tom H 2011-08-20 22:28:01 EDT
I logged on to reply to Bruno's post and noticed in Comment 6 that the UUID output by "grub2-probe --target=fs_uuid ..."  is the same for sda1, sdb2, and md0. It didn't seem correct so I booted up my VM and the output of "search --fs_uuid d926a2c0-3c34-413b-966d-7281ac8d3f7a" at the "grub" prompt is "(hd0,gpt2) (hd1,gpt2)", which confirms the grub2-probe result but blkid disagrees:

[root@localhost ~]# blkid
/dev/sda2: UUID="9f679bf8-078e-4255-3be4-632ff10ee451" UUID_SUB="3c727c1b-7302-aaa7-5d73-e73f2a2d8839" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/sda3: UUID="ea914222-ffe9-455e-a4ec-d5313fb94ab8" TYPE="swap"
/dev/sdb2: UUID="9f679bf8-078e-4255-3be4-632ff10ee451" UUID_SUB="352c117a-cdf8-3da3-4116-713b180c2394" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/sdb3: UUID="c0671e1b-f40a-4e3c-a83e-ac39f5a3b16a" TYPE="swap"
/dev/md0: UUID="d926a2c0-3c34-413b-966d-7281ac8d3f7a" TYPE="ext4"
Comment 9 Tom H 2011-08-20 22:40:01 EDT
Bruno (Comment 7)

FYI:

[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1] 
md0 : active raid1 sda2[0] sdb2[1]
      7987188 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

[root@localhost anaconda]# grep mdadm /var/log/anaconda/anaconda.program.log
19:09:22,076 INFO program: Running... mdadm --create /dev/md0 --run --level=1 --raid-devices=2 --metadata=1.0 --bitmap=internal /dev/sda2 /dev/sdb2
19:09:22,531 ERR program: mdadm: array /dev/md0 started.

My system's booting up from sda2 because (md0) isn't detected by grub and I run the following at the "grub" prompt:

set prefix=(hd0,gpt2)/boot/grub2
set root=(hd0,gpt2)
configfile /boot/grub2/grub.cfg

and select the "Fedora 3.0.0-1.fc16.i686.PAE" entry defined by

[root@localhost anaconda]# cat /etc/grub2.cfg
set timeout=9
set default=0

function load_video {
  insmod vbe
  insmod vga
  insmod video_bochs
  insmod video_cirrus
}

menuentry 'Fedora 3.0.0-1.fc16.i686.PAE' {
load_video
insmod part_gpt
#insmod raid
#insmod mdraid1x
insmod ext2
search --no-floppy --fs-uuid --set=root d926a2c0-3c34-413b-966d-7281ac8d3f7a
linux /boot/vmlinuz-3.0.0-1.fc16.i686.PAE root=UUID=d926a2c0-3c34-413b-966d-7281ac8d3f7a ro rd.lvm=0 rd.dm=0 rd.md.uuid=9f679bf8:078e4255:3be4632f:f10ee451 KEYTABLE=us SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 vga=794
initrd /boot/initramfs-3.0.0-1.fc16.i686.PAE.img
}
Comment 10 Tom H 2011-08-21 08:37:56 EDT
Since the raid modules weren't embedded in core.img by grub2-install and they weren't insmod'd by grub.cfg, I've looked at both grub2-install and grub2-mkconfig.

From "/usr/sbin/grub2-install"

# Then the partition map module.  In order to support partition-less media,
# this command is allowed to fail (--target=fs already grants us that the
# filesystem will be accessible).
partmap_module=
for x in `"$grub_probe" --device-map="${device_map}" --target=partmap --device "${grub_device}" 2> /dev/null`; do
   case "$x" in
       netbsd | openbsd) 
           partmap_module="$partmap_module part_bsd";;
       "") ;;
       *)
           partmap_module="$partmap_module part_$x";;
   esac
done

which explains why a "set" at the "grub rescue" prompt only returns "(hd0) (hd1)".

From "/usr/lib/grub/grub-mkconfig_lib

uses_abstraction () {
  device=$1

  abstraction="`${grub_probe} --device ${device} --target=abstraction`"
  for module in ${abstraction}; do
    if test "x${module}" = "x$2"; then
      return 0
    fi
  done
  return 1
}

which explains why grub2-mkconfig/10_linux don't insmod the raid modules in grub.cfg

I also noticed

# The order in this list is critical.  Be careful when modifying it.
modules="$modules $disk_module"
modules="$modules $fs_module $partmap_module $devabstraction_module"

in "/usr/sbin/grub2-install" so, without much conviction, I rebuilt core2.img and re-installed grub with grub2-setup just in case it might help; it didn't.
Comment 11 Tom H 2011-08-21 08:41:43 EDT
Slightly OT but I'm curious. Why are all the disk and filesystem "insmod" lines in grub.cfg needed when core.img is built with the necessary modules? (As I would expect it to be normally.)

I can boot with just:

set prefix=(hd0,gpt2)/boot/grub2
set root=(hd0,gpt2)
configfile /boot/grub2/grub.cfg

and

[root@localhost anaconda]# cat /etc/grub2.cfg
set timeout=9
set default=0

function load_video {
  insmod vbe
  insmod vga
  insmod video_bochs
  insmod video_cirrus
}

menuentry 'Fedora 3.0.0-1.fc16.i686.PAE' {
load_video
linux /boot/vmlinuz-3.0.0-1.fc16.i686.PAE
root=UUID=d926a2c0-3c34-413b-966d-7281ac8d3f7a ro rd.lvm=0 rd.dm=0
rd.md.uuid=9f679bf8:078e4255:3be4632f:f10ee451 KEYTABLE=us
SYSFONT=latarcyrheb-sun16 rd.luks=0 LANG=en_US.UTF-8 vga=794
initrd /boot/initramfs-3.0.0-1.fc16.i686.PAE.img
}
Comment 12 Bruno Wolff III 2011-08-21 11:13:19 EDT
You do have 1.0 metadata, so that isn't the problem. (With 1.0 metadata, the raid info is stored at the end of the block device, so that one can access an element of an array, read only, as though it were not a raid device and things will work.)
Comment 13 Tom H 2011-08-22 15:25:10 EDT
Thanks. The difference between grub1 and grub2 is that grub1 sets root as "(hdX,Y) and grub2 as "(mdZ)". Since, in the default F16 install, neither "(hd0,gpt2)" nor "(md0)" are recognized, ...

As a (possibly misinformed) user, I would've expected "grub-install /dev/sda" to install grub2 with the root device as "(hdX,Y)" like grub1 (and as Anaconda does) and "grub-install /dev/mdZ" to install grub2 with the root device as "(mdZ)".

I've re-installed grub specifying the root device with:

grub2-setup --verbose --core-image=core2.img --directory=/boot/grub2
--device-map=/boot/grub2/device.map --root-device='(hd0,gpt2)' /dev/sda
grub2-setup --verbose --core-image=core2.img --directory=/boot/grub2
--device-map=/boot/grub2/device.map --root-device='(hd0,gpt2)'  /dev/sdb

and I'm now booting to the grub menu.

Regarding the difference between metadata 0.9 and 1.0 and metadata 1.1 and 1.2, I've been intrigued since you mentioned in Comment 7, so I created an Ubuntu 11.04 VM with gpt and mdraid because it defaults to metadata 1.2.

1. I switched to the grub cli at boot and tried to boot with the same "set prefix..." and "set root" as F16; you're right, it fails. (But it works when I delete all the insmods.)

2. FYI, here's a comparison between the two VMs, should it be of interest.

Fedora

[root@localhost ~]# blkid
/dev/sda2: UUID="9f679bf8-078e-4255-3be4-632ff10ee451" UUID_SUB="3c727c1b-7302-aaa7-5d73-e73f2a2d8839" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/sdb2: UUID="9f679bf8-078e-4255-3be4-632ff10ee451" UUID_SUB="352c117a-cdf8-3da3-4116-713b180c2394" LABEL="localhost.localdomain:0" TYPE="linux_raid_member"
/dev/md0: UUID="d926a2c0-3c34-413b-966d-7281ac8d3f7a" TYPE="ext4"

[root@localhost ~]# ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root  9 Aug 22 13:33 d926a2c0-3c34-413b-966d-7281ac8d3f7a -> ../../md0

root@localhost:~# mdadm --examine /dev/sda2 | grep UUID
     Array UUID : 9f679bf8:078e4255:3be4632f:f10ee451
    Device UUID : 3c727c1b:7302aaa7:5d73e73f:2a2d8839

root@localhost:~# mdadm --examine /dev/sdb2 | grep UUID
     Array UUID : 9f679bf8:078e4255:3be4632f:f10ee451
    Device UUID : 352c117a:cdf83da3:4116713b:180c2394

[root@localhost ~]# mdadm --detail /dev/md0 | grep UUID
           UUID : 9f679bf8:078e4255:3be4632f:f10ee451

[root@localhost ~]# rpm -q --qf '%{name} %{version}-%{release}\n' util-linux
util-linux 2.19.1-2.fc16

[root@localhost ~]# rpm -q --qf '%{name} %{version}-%{release}\n' grub2
grub2 1.99-0.2.fc16

[root@localhost ~]# grub2-install --version
grub2-install (GRUB) 1.99~rc1

Ubuntu

root@localhost:~# blkid
/dev/sda2: UUID="0dae631e-c4c1-dd38-9803-d0ac0c920a79" LABEL="localhost:0" TYPE="linux_raid_member"
/dev/md0: UUID="489ac9fd-e8f8-46c9-aafb-65fc6e1db63e" TYPE="ext4"
/dev/sdb2: UUID="0dae631e-c4c1-dd38-9803-d0ac0c920a79" LABEL="localhost:0" TYPE="linux_raid_member"

root@localhost:~# ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root 9 2011-08-23 13:31 489ac9fd-e8f8-46c9-aafb-65fc6e1db63e -> ../../md0

root@localhost:~# mdadm --examine /dev/sda2 | grep UUID
     Array UUID : 0dae631e:c4c1dd38:9803d0ac:0c920a79
    Device UUID : bc8f5b99:cf735af2:e3f9fb5e:51d9ab7a

root@localhost:~# mdadm --examine /dev/sdb2 | grep UUID
     Array UUID : 0dae631e:c4c1dd38:9803d0ac:0c920a79
    Device UUID : 1afcecb8:e351edb2:cdb27a6c:5479e28b

root@localhost:~# mdadm --detail /dev/md0 | grep UUID
           UUID : 0dae631e:c4c1dd38:9803d0ac:0c920a79

root@localhost:~# dpkg-query -W -f='${Package} ${Version}\n' util-linux
util-linux 2.17.2-9.1ubuntu4

root@localhost:~# dpkg-query -W -f='${Package} ${Version}\n' grub-pc
grub-pc 1.99~rc1-13ubuntu3

root@localhost:~# grub-install --version
grub-install (GRUB) 1.99~rc1-13ubuntu3
Comment 14 Tom H 2011-08-22 22:08:36 EDT
I duped my VM then downloaded and installed ftp://ftp.gnu.org/gnu/grub/grub-1.99.tar.gz.

It boots OK.

[root@localhost ~]# grub-probe / --target=device
/dev/md0

[root@localhost ~]# grub-probe / --target=drive
(mduuid/9f679bf8078e42553be4632ff10ee451)

[root@localhost ~]# grub-probe --device /dev/md0 --target=drive
(mduuid/9f679bf8078e42553be4632ff10ee451)

When I switch to the "grub" prompt at boot

ls outputs:
(md/0) <...all disks and partitions>

set outputs
prefix=(mduuid/9f679bf8078e42553be4632ff10ee451)/boot/grub
root=(mduuid/9f679bf8078e42553be4632ff10ee451)

FYI:

[root@localhost ~]# cat /boot/grub/device.map
(hd0)	/dev/disk/by-id/ata-VBOX_HARDDISK_VB1fd4a442-c04e76f8
(hd1)	/dev/disk/by-id/ata-VBOX_HARDDISK_VBc3c46fad-843f828f
Comment 15 Tom H 2011-09-02 05:44:23 EDT
grub2-1.99-1.fc16 has resolved the issue. Thanks.
Comment 16 Fedora Admin XMLRPC Client 2011-09-16 15:08:09 EDT
This package has changed ownership in the Fedora Package Database.  Reassigning to the new owner of this component.
Comment 17 Jerry Amundson 2011-09-23 21:40:43 EDT
(In reply to comment #0)
> Description of problem:
> The installation of F16 on an MDRAID array succeeds but rebooting into the new
> install drops you to a "grub rescue" prompt.

Me too.

*** This bug has been marked as a duplicate of bug 732164 ***

Note You need to log in before you can comment on or make changes to this bug.