Bug 1201962 - Fails to assemble root RAID set specified in /etc/mdadm.conf
Summary: Fails to assemble root RAID set specified in /etc/mdadm.conf
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: dracut
Version: 22
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: dracut-maint-list
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1223017 1229289 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-14 01:45 UTC by Adam Williamson
Modified: 2019-11-08 18:59 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-19 19:32:59 UTC


Attachments (Terms of Use)
log of an affected boot with rd.debug (including the manual messing around later when I assembled the set) (1.89 MB, text/plain)
2015-03-14 01:45 UTC, Adam Williamson
no flags Details

Description Adam Williamson 2015-03-14 01:45:57 UTC
Created attachment 1001541 [details]
log of an affected boot with rd.debug (including the manual messing around later when I assembled the set)

Recently I moved my entire Fedora install from a single disk to a RAID-5 set in the same LVM VG.

Upon reboot, the system failed to boot, because dracut did not bring up the RAID set. If I add rd.auto to the cmdline, the system boots successfully.

However, the RAID set is listed in /etc/mdadm.conf:

[root@adam adamw]# cat /etc/mdadm.conf 
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md126 metadata=1.2 spares=1 name=adam.happyassassin.net:126 UUID=3475129d:15fa3455:59f9c1dc:ca05d1e7

and mdadm --assemble --scan works to mount it, indicating that the config file entry is good. I do not have any kind of rd.md.conf or similar in my cmdline, so AIUI, dracut is supposed to activate the array listed in the config file.

The /etc/mdadm.conf file is present in the initramfs; in fact I ran the 'mdadm --assemble --scan' test from the dracut shell (after booting with rd.shell) to verify that dracut *could* assemble the set if it was inclined to do so.

I will attach a log of an affected boot, with 'rd.debug'.

Comment 1 Harald Hoyer 2015-03-16 10:48:25 UTC
If you change your hardware configuration, you should:

- boot the changed system with the rescue initramfs and "rd.auto" on the kernel command line
- re-create the initramfs with:
# dracut --regenerate-all --force

Comment 2 Harald Hoyer 2015-03-16 10:49:14 UTC
and also make sure you have the proposed kernel command line parameter of:

# dracut --print-cmdline

Comment 3 Adam Williamson 2015-03-16 15:37:42 UTC
I did regenerate the initramfs; the mdraid tools and /etc/mdadm.conf file were included in the newly-generated initramfs. I still believe it is a bug that the initramfs does not assemble a RAID set specified in the /etc/mdadm.conf file that is present in its environment. All indications I could find in the documentation are that it should. As I said, from a dracut shell, I could run 'mdadm --assemble --scan' manually - the effect of which is 'assemble RAID sets found in /etc/mdadm.conf' - and that worked.

I can make it work in various ways - adding rd.auto to the command line or md.raid.uuid (or whatever that one's called exactly). Me being able to make it work isn't the issue. dracut not acting as (apparently) advertised/intended is the issue.

Comment 4 Harald Hoyer 2015-03-26 16:18:38 UTC
We do not want to auto assemble all raids we find. People have crazy /etc/mdadm.conf, so we only assemble what we are told to.

Either:
- nothing, by default
- rd.md.uuid=... only the specific raids with the uuid
- rd.auto everything we find (also lvm and dmraid and stuff)

Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted

Comment 5 Adam Williamson 2015-03-26 16:22:04 UTC
This is really not at all clear from the docs; if the default is not to detect any sets or use mdadm.conf , what is the point of these options documented in 'man dracut.cmdline':

       rd.md=0
           disable MD RAID detection

       rd.md.conf=0
           ignore mdadm.conf included in initramfs

Comment 6 Harald Hoyer 2015-03-26 16:26:50 UTC
(In reply to awilliam@redhat.com from comment #5)
> This is really not at all clear from the docs; if the default is not to
> detect any sets or use mdadm.conf , what is the point of these options
> documented in 'man dracut.cmdline':
> 
>        rd.md=0
>            disable MD RAID detection

If rd.md.uuid=... config is stored in the initramfs (/etc/cmdline.d) this is the switch to turn it off.

> 
>        rd.md.conf=0
>            ignore mdadm.conf included in initramfs

If mdadm.conf contains old/broken information "rd.md.conf=0" removes /etc/mdadm.conf stored in the initramfs.

Comment 7 Thomas Moschny 2015-05-20 16:17:37 UTC
*** Bug 1223017 has been marked as a duplicate of this bug. ***

Comment 8 Thomas Moschny 2015-05-20 16:41:03 UTC
To add a datapoint, having rd.md.uuid=... in the dracut cmdline (as printed by dracut --print-cmdline) is not enough. I had to add rd.auto=1 to grub2.cfg.

Is that intended/expected?

Comment 9 David Woodhouse 2015-06-02 17:04:22 UTC
I have a similar problem to the one described in bug 1223017. After upgrading from Fedora 20, my MD arrays are no longer automatically assembled.

For me, just adding rd.md.uuid=4ec0bafa:acd9dd7f:933cad26:7eb047ee to the command line seems to have made it boot.

Interestingly, the grub2.cfg file did contain that UUID:

        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/20166bf124f712c057559a006d37fb89'  3a9109c0-9ea7-4afa-b2e2-e8758335614b

... but it was missing from the kernel command line.

Comment 10 David Woodhouse 2015-06-02 17:05:42 UTC
ISTR this *also* broke when I upgraded this machine *to* Fedora 20. Some better QA here would be much appreciated.

Comment 11 David Woodhouse 2015-06-02 18:26:27 UTC
(In reply to David Woodhouse from comment #9)

oops, that wasn't the same UUID. But grub2.cfg *does* contain the right UUID; it's just not in a particular boot target. It's further up:

set root='mduuid/4ec0bafaacd9dd7f933cad267eb047ee'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint='mduuid/4ec0bafaacd9dd7f933cad267eb047ee'  bf1c0527-959f-4ab2-a3b0-cb75452ea5ff
else
  search --no-floppy --fs-uuid --set=root bf1c0527-959f-4ab2-a3b0-cb75452ea5ff
fi

Comment 12 Eric Bakkum 2015-06-08 13:07:31 UTC
*** Bug 1229289 has been marked as a duplicate of this bug. ***

Comment 13 Harald Hoyer 2015-06-16 13:03:52 UTC
(In reply to Thomas Moschny from comment #8)
> To add a datapoint, having rd.md.uuid=... in the dracut cmdline (as printed
> by dracut --print-cmdline) is not enough. I had to add rd.auto=1 to
> grub2.cfg.
> 
> Is that intended/expected?

Can you give me more data? What does dracut print? What does the rdsosreport.txt contain?

Comment 14 Jeff Buhrt 2015-07-01 19:36:03 UTC
1) Default F22 fedup fails with LVM on top of MD RAID (dos partitioning table with RAID /boot fails) but a machine with gpt boot upgraded fine.

A laptop w/o any RAID upgraded 'fine' [Other than a couple very annoying problems I need to make some time to log. Ex: a new request to the sound device changes the volume to 100% for the new sound, also lldpad's SELinux failures ]

fedup --network 22

reboot, fails & drops to shell.

Manually start MDRAID & LVM:
mdadm --examine --scan >> /etc/mdadm.conf
mdadm -A -s
cat /proc/mdstat

lvm vgscan
lvm vgchange -a y
lvm lvs

^D

I see the hotdog flash by, reboots, but still on F21 vs F22


2) Added md more explictly
fedup --network 22

In /boot/grub2/grub.cfg added:
rd.auto rd.md.auto
before: domdadm dolvm

reboot... finds partitions (MD+LVM+mounts are fine).
But fedup to 22 didn't do the upgrade
The fedup.log looks like it updated, but nothing was changed

Also the internal kernel command line from initramfs only had the domdadm dolvm and said it disabled the MD RAID.

I also tested without the initramfs and get 'personality for level 1 is not loaded!'. A test remaking the initramfs does the same thing (but doesn't panic like removing the initramfs from grub.cfg did.)

----
Fdisk /dev/sda on failing F21->F22 (for the failing machine)
Command (m for help): p
Disk /dev/sda: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x51f0fe24

Device     Boot     Start        End    Sectors   Size Id Type
/dev/sda1  *         2048     208844     206797   101M fd Linux raid autodetect
/dev/sda2          208845  390732929  390524085 186.2G fd Linux raid autodetect
/dev/sda3       390732930  976768064  586035135 279.5G fd Linux raid autodetect
/dev/sda4       976768065 2930272064 1953504000 931.5G fd Linux raid autodetect

Example machine that worked fine:
Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 2A4A1BFE-BEE1-428C-ACD0-60620206E00D

Device          Start        End    Sectors   Size Type
/dev/sda1        2048       4095       2048     1M BIOS boot
/dev/sda3        6144 3906254847 3906248704   1.8T Linux RAID
/dev/sda4  3906254848 5860533134 1954278287 931.9G Linux RAID


In addition to reviewing open Fedora bugs I also noticed:
http://grokbase.com/t/centos/centos-devel/14736h5p0x/upgrade-tool-patching-and-debugging

/var/log/fedup.log's:
https://www.dropbox.com/s/024poohvadopvju/fedup.log.20150701-1056?dl=0
https://www.dropbox.com/s/ni6gaog7fwm45p4/fedup.log.20150701-1253?dl=0

Comment 15 Dusty Mabe 2015-07-04 22:01:38 UTC
I am seeing this as well. I have a very complicated setup (mdraid/crypt/lvm/) but I have the exact same setup on F21 as I am attempting for F22 and booting into the system works just fine without any workarounds:

kernel command line for F21:
$ cat /etc/system-release
Fedora release 21 (Twenty One)
$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/vgroot/lvroot ro


In F22 with the same setup (fresh install) I have to add rd.auto to the cmdline to get it to work:

# cat /etc/system-release 
Fedora release 22 (Twenty Two)
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/mapper/vgroot-lvroot ro rd.auto


Any changes between F21/F22 that would have caused this regression?

Comment 16 sharif-redhat 2015-07-06 17:34:33 UTC
Confirmed workaround:
Adding "rd.md.uuid=<uuid of /boot md vol> rd.md.uuid=<uuid of / md vol>"
to GRUB_CMDLINE_LINUX in /etc/default/grub and regenerating grub.cfg.

Comment 17 Pekka Savola 2015-08-03 08:41:16 UTC
I was also bitten by this when upgrading to F22. Regenerating initramfs doesn't help. Rd.auto works fine. As would likely hardcoding the UUIDs, but I don't want to do this.

First, a year or two ago persistent names stopped working (/dev/md0 etc.). Working as designed, they said.

OK, I changed the software raid config to use labels, because I didn't want to hardcode UUID's to grub config, fstab, etc. So the grub config is something like this:

title Fedora (4.1.2-200.fc22.i686+PAE) 22 (Twenty Two)
        root (hd0,0)
        kernel /vmlinuz-4.1.2-200.fc22.i686+PAE root=LABEL=Root quiet SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=fi-latin1 rw rd.info rd.convertfs enforcing=0
        initrd /initramfs-4.1.2-200.fc22.i686+PAE.img

Now this has stopped working as well. Again working as designed, right? Sounds like badly designed to me.

Comment 18 Eric Bakkum 2015-08-04 12:38:56 UTC
If I understand correctly with respect to RAID, in previous Fedora versions rd.md.uuid etc used to be stored in the initramfs in /etc/cmdline.d/90mdraid.conf, so it was not required to specify them explicitly on the kernel command line in grub.cfg. From comment 4 I understand that it is not considered good practice to store configuration dependent settings in the initramfs, and as of Fedora 22 dracut does not generate the 90mdraid.conf anymore by default.

The rd.md.uuid now have to be supplied on the kernel command line explicitly. The fresh Fedora 22 installation seems to take care of that, but Fedup not always seems to do so. That is at least what I think that got me stuck, until I found this bugreport (see https://bugzilla.redhat.com/show_bug.cgi?id=1229289#c4).

Apart from adding the rd.md.uuid in grub.cfg, I also found out that you can go back to the pre-Fedora22 situation and have dracut include the /etc/cmdline.d/90mdraid.conf by setting hostonly_cmdline="yes" in /usr/lib/dracut/dracut.conf.d/01-dist.conf . It helped out in my case, but I am not sure if this works for everybody, and assume that this is not the advised procedure anyhow.

Comment 19 srakitnican 2015-09-03 08:45:33 UTC
Hi(In reply to Harald Hoyer from comment #4)
> We do not want to auto assemble all raids we find. People have crazy
> /etc/mdadm.conf, so we only assemble what we are told to.
> 
> Either:
> - nothing, by default
> - rd.md.uuid=... only the specific raids with the uuid
> - rd.auto everything we find (also lvm and dmraid and stuff)
> 
> Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted


Just an idea... is it possible to detect and add/modify rd.md.uuid with grub2-mkconfig from currently mounted / system?

Comment 20 Harald Hoyer 2015-09-03 10:06:22 UTC
(In reply to semiRocket from comment #19)
> Hi(In reply to Harald Hoyer from comment #4)
> > We do not want to auto assemble all raids we find. People have crazy
> > /etc/mdadm.conf, so we only assemble what we are told to.
> > 
> > Either:
> > - nothing, by default
> > - rd.md.uuid=... only the specific raids with the uuid
> > - rd.auto everything we find (also lvm and dmraid and stuff)
> > 
> > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted
> 
> 
> Just an idea... is it possible to detect and add/modify rd.md.uuid with
> grub2-mkconfig from currently mounted / system?

sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods dracut uses, or just use the values from "dracut --print-cmdline"

Comment 21 srakitnican 2015-09-03 10:28:19 UTC
(In reply to Harald Hoyer from comment #20)
> (In reply to semiRocket from comment #19)
> > Hi(In reply to Harald Hoyer from comment #4)
> > > We do not want to auto assemble all raids we find. People have crazy
> > > /etc/mdadm.conf, so we only assemble what we are told to.
> > > 
> > > Either:
> > > - nothing, by default
> > > - rd.md.uuid=... only the specific raids with the uuid
> > > - rd.auto everything we find (also lvm and dmraid and stuff)
> > > 
> > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted
> > 
> > 
> > Just an idea... is it possible to detect and add/modify rd.md.uuid with
> > grub2-mkconfig from currently mounted / system?
> 
> sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods
> dracut uses, or just use the values from "dracut --print-cmdline"

Sorry I was not expressed correctly, I know grub2-mkconfig remakes based on configuration, but, is it possible that for example dracut update GRUB_CMDLINE_LINUX in /etc/default/grub based on currently mounted / system for  example from chroot env:

# mount /dev/md126p2 /mnt
# for d in /sys /dev /run /proc ; do mount -v --bind "$d" /mnt"$d" ; done
# chroot /mnt
# dracut -v --force --regenerate-all

And dracut here updates grub cmdline with / rd.md.uuid

Maybe I am to picky, but was expecting that or similar behavior.

Comment 22 Harald Hoyer 2015-09-03 10:58:29 UTC
(In reply to semiRocket from comment #21)
> (In reply to Harald Hoyer from comment #20)
> > (In reply to semiRocket from comment #19)
> > > Hi(In reply to Harald Hoyer from comment #4)
> > > > We do not want to auto assemble all raids we find. People have crazy
> > > > /etc/mdadm.conf, so we only assemble what we are told to.
> > > > 
> > > > Either:
> > > > - nothing, by default
> > > > - rd.md.uuid=... only the specific raids with the uuid
> > > > - rd.auto everything we find (also lvm and dmraid and stuff)
> > > > 
> > > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted
> > > 
> > > 
> > > Just an idea... is it possible to detect and add/modify rd.md.uuid with
> > > grub2-mkconfig from currently mounted / system?
> > 
> > sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods
> > dracut uses, or just use the values from "dracut --print-cmdline"
> 
> Sorry I was not expressed correctly, I know grub2-mkconfig remakes based on
> configuration, but, is it possible that for example dracut update
> GRUB_CMDLINE_LINUX in /etc/default/grub based on currently mounted / system
> for  example from chroot env:
> 
> # mount /dev/md126p2 /mnt
> # for d in /sys /dev /run /proc ; do mount -v --bind "$d" /mnt"$d" ; done
> # chroot /mnt
> # dracut -v --force --regenerate-all
> 
> And dracut here updates grub cmdline with / rd.md.uuid
> 
> Maybe I am to picky, but was expecting that or similar behavior.

dracut will not get in the business of configuring grub. All it will do is generate an initramfs.

Please open a bug against "grubby" or grub2-mkconfig, if you want them to take this into account.

Comment 23 srakitnican 2015-09-03 18:59:01 UTC
Here it is: https://bugzilla.redhat.com/show_bug.cgi?id=1259911

Comment 24 David Woodhouse 2015-09-04 13:44:27 UTC
I'm a little confused. Why open a new bug instead of just reassigning this one to grubby or grub2-mkconfig?

Comment 25 Adam Williamson 2016-03-03 22:40:44 UTC
well, I just updated my F24 system, rebooted, and suddenly needed rd.auto again or it wouldn't boot. Dunno what the hell broke THIS time.

Comment 26 Robin 2016-05-21 22:30:29 UTC
I'm having problems with F24 as well but for me rd.auto=1 doesn't help. I can boot into rescue mode from the installer and assemble the array but even after rebuilding initramfs it's still not working.

Comment 27 Dusty Mabe 2016-07-12 17:24:09 UTC
(In reply to Dusty Mabe from comment #15)
> I am seeing this as well. I have a very complicated setup
> (mdraid/crypt/lvm/) but I have the exact same setup on F21 as I am
> attempting for F22 and booting into the system works just fine without any
> workarounds:
> 
> kernel command line for F21:
> $ cat /etc/system-release
> Fedora release 21 (Twenty One)
> $ cat /proc/cmdline 
> BOOT_IMAGE=/boot/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/vgroot/lvroot ro
> 
> 
> In F22 with the same setup (fresh install) I have to add rd.auto to the
> cmdline to get it to work:
> 
> # cat /etc/system-release 
> Fedora release 22 (Twenty Two)
> $ cat /proc/cmdline
> BOOT_IMAGE=/boot/vmlinuz-4.0.4-301.fc22.x86_64
> root=/dev/mapper/vgroot-lvroot ro rd.auto
> 
> 
> Any changes between F21/F22 that would have caused this regression?



Finally upgraded my computer from F22 to F24 and same issue. Just would like to ask again: Any changes after 21 that would have caused this regression? Was it by design?

Comment 28 srakitnican 2016-07-12 19:27:16 UTC
(In reply to Dusty Mabe from comment #27)
> Finally upgraded my computer from F22 to F24 and same issue. Just would like
> to ask again: Any changes after 21 that would have caused this regression?
> Was it by design?

Is comment #4 not what you are looking for?

The way I understand is that they changed it to not assemble any array on boot because there were complains about it. If raid setup is present, currently  /etc/default/grub has to be edited with rd.md.uuid to assemble.

Don't know if anaconda configures that on new installs. Judging by your experience it seems not.

Comment 29 Dusty Mabe 2016-07-12 19:42:16 UTC
(In reply to srakitnican from comment #28)
> (In reply to Dusty Mabe from comment #27)
> > Finally upgraded my computer from F22 to F24 and same issue. Just would like
> > to ask again: Any changes after 21 that would have caused this regression?
> > Was it by design?
> 
> Is comment #4 not what you are looking for?

you are right. should this bug be closed then? 

> 
> The way I understand is that they changed it to not assemble any array on
> boot because there were complains about it. If raid setup is present,
> currently  /etc/default/grub has to be edited with rd.md.uuid to assemble.
> 
> Don't know if anaconda configures that on new installs. Judging by your
> experience it seems not.

I use a custom setup built from scratch so I don't think we can make that statement based on my experience.

Comment 30 Adam Williamson 2016-07-12 19:48:26 UTC
adding rd.auto still works too, if you're lazy. Not sure if there's any reason to keep the bug open...

Comment 31 Fedora End Of Life 2016-07-19 19:32:59 UTC
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 32 Harald Hoyer 2016-08-22 09:36:31 UTC
please double check your grub config and your kernel command line parameter with what dracut suggests:

$ sudo dracut --print-cmdline


Note You need to log in before you can comment on or make changes to this bug.