Created attachment 1001541 [details] log of an affected boot with rd.debug (including the manual messing around later when I assembled the set) Recently I moved my entire Fedora install from a single disk to a RAID-5 set in the same LVM VG. Upon reboot, the system failed to boot, because dracut did not bring up the RAID set. If I add rd.auto to the cmdline, the system boots successfully. However, the RAID set is listed in /etc/mdadm.conf: [root@adam adamw]# cat /etc/mdadm.conf # mdadm.conf written out by anaconda MAILADDR root AUTO +imsm +1.x -all ARRAY /dev/md126 metadata=1.2 spares=1 name=adam.happyassassin.net:126 UUID=3475129d:15fa3455:59f9c1dc:ca05d1e7 and mdadm --assemble --scan works to mount it, indicating that the config file entry is good. I do not have any kind of rd.md.conf or similar in my cmdline, so AIUI, dracut is supposed to activate the array listed in the config file. The /etc/mdadm.conf file is present in the initramfs; in fact I ran the 'mdadm --assemble --scan' test from the dracut shell (after booting with rd.shell) to verify that dracut *could* assemble the set if it was inclined to do so. I will attach a log of an affected boot, with 'rd.debug'.
If you change your hardware configuration, you should: - boot the changed system with the rescue initramfs and "rd.auto" on the kernel command line - re-create the initramfs with: # dracut --regenerate-all --force
and also make sure you have the proposed kernel command line parameter of: # dracut --print-cmdline
I did regenerate the initramfs; the mdraid tools and /etc/mdadm.conf file were included in the newly-generated initramfs. I still believe it is a bug that the initramfs does not assemble a RAID set specified in the /etc/mdadm.conf file that is present in its environment. All indications I could find in the documentation are that it should. As I said, from a dracut shell, I could run 'mdadm --assemble --scan' manually - the effect of which is 'assemble RAID sets found in /etc/mdadm.conf' - and that worked. I can make it work in various ways - adding rd.auto to the command line or md.raid.uuid (or whatever that one's called exactly). Me being able to make it work isn't the issue. dracut not acting as (apparently) advertised/intended is the issue.
We do not want to auto assemble all raids we find. People have crazy /etc/mdadm.conf, so we only assemble what we are told to. Either: - nothing, by default - rd.md.uuid=... only the specific raids with the uuid - rd.auto everything we find (also lvm and dmraid and stuff) Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted
This is really not at all clear from the docs; if the default is not to detect any sets or use mdadm.conf , what is the point of these options documented in 'man dracut.cmdline': rd.md=0 disable MD RAID detection rd.md.conf=0 ignore mdadm.conf included in initramfs
(In reply to awilliam from comment #5) > This is really not at all clear from the docs; if the default is not to > detect any sets or use mdadm.conf , what is the point of these options > documented in 'man dracut.cmdline': > > rd.md=0 > disable MD RAID detection If rd.md.uuid=... config is stored in the initramfs (/etc/cmdline.d) this is the switch to turn it off. > > rd.md.conf=0 > ignore mdadm.conf included in initramfs If mdadm.conf contains old/broken information "rd.md.conf=0" removes /etc/mdadm.conf stored in the initramfs.
*** Bug 1223017 has been marked as a duplicate of this bug. ***
To add a datapoint, having rd.md.uuid=... in the dracut cmdline (as printed by dracut --print-cmdline) is not enough. I had to add rd.auto=1 to grub2.cfg. Is that intended/expected?
I have a similar problem to the one described in bug 1223017. After upgrading from Fedora 20, my MD arrays are no longer automatically assembled. For me, just adding rd.md.uuid=4ec0bafa:acd9dd7f:933cad26:7eb047ee to the command line seems to have made it boot. Interestingly, the grub2.cfg file did contain that UUID: if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/20166bf124f712c057559a006d37fb89' 3a9109c0-9ea7-4afa-b2e2-e8758335614b ... but it was missing from the kernel command line.
ISTR this *also* broke when I upgraded this machine *to* Fedora 20. Some better QA here would be much appreciated.
(In reply to David Woodhouse from comment #9) oops, that wasn't the same UUID. But grub2.cfg *does* contain the right UUID; it's just not in a particular boot target. It's further up: set root='mduuid/4ec0bafaacd9dd7f933cad267eb047ee' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/4ec0bafaacd9dd7f933cad267eb047ee' bf1c0527-959f-4ab2-a3b0-cb75452ea5ff else search --no-floppy --fs-uuid --set=root bf1c0527-959f-4ab2-a3b0-cb75452ea5ff fi
*** Bug 1229289 has been marked as a duplicate of this bug. ***
(In reply to Thomas Moschny from comment #8) > To add a datapoint, having rd.md.uuid=... in the dracut cmdline (as printed > by dracut --print-cmdline) is not enough. I had to add rd.auto=1 to > grub2.cfg. > > Is that intended/expected? Can you give me more data? What does dracut print? What does the rdsosreport.txt contain?
1) Default F22 fedup fails with LVM on top of MD RAID (dos partitioning table with RAID /boot fails) but a machine with gpt boot upgraded fine. A laptop w/o any RAID upgraded 'fine' [Other than a couple very annoying problems I need to make some time to log. Ex: a new request to the sound device changes the volume to 100% for the new sound, also lldpad's SELinux failures ] fedup --network 22 reboot, fails & drops to shell. Manually start MDRAID & LVM: mdadm --examine --scan >> /etc/mdadm.conf mdadm -A -s cat /proc/mdstat lvm vgscan lvm vgchange -a y lvm lvs ^D I see the hotdog flash by, reboots, but still on F21 vs F22 2) Added md more explictly fedup --network 22 In /boot/grub2/grub.cfg added: rd.auto rd.md.auto before: domdadm dolvm reboot... finds partitions (MD+LVM+mounts are fine). But fedup to 22 didn't do the upgrade The fedup.log looks like it updated, but nothing was changed Also the internal kernel command line from initramfs only had the domdadm dolvm and said it disabled the MD RAID. I also tested without the initramfs and get 'personality for level 1 is not loaded!'. A test remaking the initramfs does the same thing (but doesn't panic like removing the initramfs from grub.cfg did.) ---- Fdisk /dev/sda on failing F21->F22 (for the failing machine) Command (m for help): p Disk /dev/sda: 1.4 TiB, 1500301910016 bytes, 2930277168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x51f0fe24 Device Boot Start End Sectors Size Id Type /dev/sda1 * 2048 208844 206797 101M fd Linux raid autodetect /dev/sda2 208845 390732929 390524085 186.2G fd Linux raid autodetect /dev/sda3 390732930 976768064 586035135 279.5G fd Linux raid autodetect /dev/sda4 976768065 2930272064 1953504000 931.5G fd Linux raid autodetect Example machine that worked fine: Disk /dev/sda: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 2A4A1BFE-BEE1-428C-ACD0-60620206E00D Device Start End Sectors Size Type /dev/sda1 2048 4095 2048 1M BIOS boot /dev/sda3 6144 3906254847 3906248704 1.8T Linux RAID /dev/sda4 3906254848 5860533134 1954278287 931.9G Linux RAID In addition to reviewing open Fedora bugs I also noticed: http://grokbase.com/t/centos/centos-devel/14736h5p0x/upgrade-tool-patching-and-debugging /var/log/fedup.log's: https://www.dropbox.com/s/024poohvadopvju/fedup.log.20150701-1056?dl=0 https://www.dropbox.com/s/ni6gaog7fwm45p4/fedup.log.20150701-1253?dl=0
I am seeing this as well. I have a very complicated setup (mdraid/crypt/lvm/) but I have the exact same setup on F21 as I am attempting for F22 and booting into the system works just fine without any workarounds: kernel command line for F21: $ cat /etc/system-release Fedora release 21 (Twenty One) $ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/vgroot/lvroot ro In F22 with the same setup (fresh install) I have to add rd.auto to the cmdline to get it to work: # cat /etc/system-release Fedora release 22 (Twenty Two) $ cat /proc/cmdline BOOT_IMAGE=/boot/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/mapper/vgroot-lvroot ro rd.auto Any changes between F21/F22 that would have caused this regression?
Confirmed workaround: Adding "rd.md.uuid=<uuid of /boot md vol> rd.md.uuid=<uuid of / md vol>" to GRUB_CMDLINE_LINUX in /etc/default/grub and regenerating grub.cfg.
I was also bitten by this when upgrading to F22. Regenerating initramfs doesn't help. Rd.auto works fine. As would likely hardcoding the UUIDs, but I don't want to do this. First, a year or two ago persistent names stopped working (/dev/md0 etc.). Working as designed, they said. OK, I changed the software raid config to use labels, because I didn't want to hardcode UUID's to grub config, fstab, etc. So the grub config is something like this: title Fedora (4.1.2-200.fc22.i686+PAE) 22 (Twenty Two) root (hd0,0) kernel /vmlinuz-4.1.2-200.fc22.i686+PAE root=LABEL=Root quiet SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=fi-latin1 rw rd.info rd.convertfs enforcing=0 initrd /initramfs-4.1.2-200.fc22.i686+PAE.img Now this has stopped working as well. Again working as designed, right? Sounds like badly designed to me.
If I understand correctly with respect to RAID, in previous Fedora versions rd.md.uuid etc used to be stored in the initramfs in /etc/cmdline.d/90mdraid.conf, so it was not required to specify them explicitly on the kernel command line in grub.cfg. From comment 4 I understand that it is not considered good practice to store configuration dependent settings in the initramfs, and as of Fedora 22 dracut does not generate the 90mdraid.conf anymore by default. The rd.md.uuid now have to be supplied on the kernel command line explicitly. The fresh Fedora 22 installation seems to take care of that, but Fedup not always seems to do so. That is at least what I think that got me stuck, until I found this bugreport (see https://bugzilla.redhat.com/show_bug.cgi?id=1229289#c4). Apart from adding the rd.md.uuid in grub.cfg, I also found out that you can go back to the pre-Fedora22 situation and have dracut include the /etc/cmdline.d/90mdraid.conf by setting hostonly_cmdline="yes" in /usr/lib/dracut/dracut.conf.d/01-dist.conf . It helped out in my case, but I am not sure if this works for everybody, and assume that this is not the advised procedure anyhow.
Hi(In reply to Harald Hoyer from comment #4) > We do not want to auto assemble all raids we find. People have crazy > /etc/mdadm.conf, so we only assemble what we are told to. > > Either: > - nothing, by default > - rd.md.uuid=... only the specific raids with the uuid > - rd.auto everything we find (also lvm and dmraid and stuff) > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted Just an idea... is it possible to detect and add/modify rd.md.uuid with grub2-mkconfig from currently mounted / system?
(In reply to semiRocket from comment #19) > Hi(In reply to Harald Hoyer from comment #4) > > We do not want to auto assemble all raids we find. People have crazy > > /etc/mdadm.conf, so we only assemble what we are told to. > > > > Either: > > - nothing, by default > > - rd.md.uuid=... only the specific raids with the uuid > > - rd.auto everything we find (also lvm and dmraid and stuff) > > > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted > > > Just an idea... is it possible to detect and add/modify rd.md.uuid with > grub2-mkconfig from currently mounted / system? sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods dracut uses, or just use the values from "dracut --print-cmdline"
(In reply to Harald Hoyer from comment #20) > (In reply to semiRocket from comment #19) > > Hi(In reply to Harald Hoyer from comment #4) > > > We do not want to auto assemble all raids we find. People have crazy > > > /etc/mdadm.conf, so we only assemble what we are told to. > > > > > > Either: > > > - nothing, by default > > > - rd.md.uuid=... only the specific raids with the uuid > > > - rd.auto everything we find (also lvm and dmraid and stuff) > > > > > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted > > > > > > Just an idea... is it possible to detect and add/modify rd.md.uuid with > > grub2-mkconfig from currently mounted / system? > > sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods > dracut uses, or just use the values from "dracut --print-cmdline" Sorry I was not expressed correctly, I know grub2-mkconfig remakes based on configuration, but, is it possible that for example dracut update GRUB_CMDLINE_LINUX in /etc/default/grub based on currently mounted / system for example from chroot env: # mount /dev/md126p2 /mnt # for d in /sys /dev /run /proc ; do mount -v --bind "$d" /mnt"$d" ; done # chroot /mnt # dracut -v --force --regenerate-all And dracut here updates grub cmdline with / rd.md.uuid Maybe I am to picky, but was expecting that or similar behavior.
(In reply to semiRocket from comment #21) > (In reply to Harald Hoyer from comment #20) > > (In reply to semiRocket from comment #19) > > > Hi(In reply to Harald Hoyer from comment #4) > > > > We do not want to auto assemble all raids we find. People have crazy > > > > /etc/mdadm.conf, so we only assemble what we are told to. > > > > > > > > Either: > > > > - nothing, by default > > > > - rd.md.uuid=... only the specific raids with the uuid > > > > - rd.auto everything we find (also lvm and dmraid and stuff) > > > > > > > > Maybe I could add rd.md.auto as "mdadm --assemble --scan" if wanted > > > > > > > > > Just an idea... is it possible to detect and add/modify rd.md.uuid with > > > grub2-mkconfig from currently mounted / system? > > > > sure.. grub2-mkconfig has script plugins, AFAIK... either use the methods > > dracut uses, or just use the values from "dracut --print-cmdline" > > Sorry I was not expressed correctly, I know grub2-mkconfig remakes based on > configuration, but, is it possible that for example dracut update > GRUB_CMDLINE_LINUX in /etc/default/grub based on currently mounted / system > for example from chroot env: > > # mount /dev/md126p2 /mnt > # for d in /sys /dev /run /proc ; do mount -v --bind "$d" /mnt"$d" ; done > # chroot /mnt > # dracut -v --force --regenerate-all > > And dracut here updates grub cmdline with / rd.md.uuid > > Maybe I am to picky, but was expecting that or similar behavior. dracut will not get in the business of configuring grub. All it will do is generate an initramfs. Please open a bug against "grubby" or grub2-mkconfig, if you want them to take this into account.
Here it is: https://bugzilla.redhat.com/show_bug.cgi?id=1259911
I'm a little confused. Why open a new bug instead of just reassigning this one to grubby or grub2-mkconfig?
well, I just updated my F24 system, rebooted, and suddenly needed rd.auto again or it wouldn't boot. Dunno what the hell broke THIS time.
I'm having problems with F24 as well but for me rd.auto=1 doesn't help. I can boot into rescue mode from the installer and assemble the array but even after rebuilding initramfs it's still not working.
(In reply to Dusty Mabe from comment #15) > I am seeing this as well. I have a very complicated setup > (mdraid/crypt/lvm/) but I have the exact same setup on F21 as I am > attempting for F22 and booting into the system works just fine without any > workarounds: > > kernel command line for F21: > $ cat /etc/system-release > Fedora release 21 (Twenty One) > $ cat /proc/cmdline > BOOT_IMAGE=/boot/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/vgroot/lvroot ro > > > In F22 with the same setup (fresh install) I have to add rd.auto to the > cmdline to get it to work: > > # cat /etc/system-release > Fedora release 22 (Twenty Two) > $ cat /proc/cmdline > BOOT_IMAGE=/boot/vmlinuz-4.0.4-301.fc22.x86_64 > root=/dev/mapper/vgroot-lvroot ro rd.auto > > > Any changes between F21/F22 that would have caused this regression? Finally upgraded my computer from F22 to F24 and same issue. Just would like to ask again: Any changes after 21 that would have caused this regression? Was it by design?
(In reply to Dusty Mabe from comment #27) > Finally upgraded my computer from F22 to F24 and same issue. Just would like > to ask again: Any changes after 21 that would have caused this regression? > Was it by design? Is comment #4 not what you are looking for? The way I understand is that they changed it to not assemble any array on boot because there were complains about it. If raid setup is present, currently /etc/default/grub has to be edited with rd.md.uuid to assemble. Don't know if anaconda configures that on new installs. Judging by your experience it seems not.
(In reply to srakitnican from comment #28) > (In reply to Dusty Mabe from comment #27) > > Finally upgraded my computer from F22 to F24 and same issue. Just would like > > to ask again: Any changes after 21 that would have caused this regression? > > Was it by design? > > Is comment #4 not what you are looking for? you are right. should this bug be closed then? > > The way I understand is that they changed it to not assemble any array on > boot because there were complains about it. If raid setup is present, > currently /etc/default/grub has to be edited with rd.md.uuid to assemble. > > Don't know if anaconda configures that on new installs. Judging by your > experience it seems not. I use a custom setup built from scratch so I don't think we can make that statement based on my experience.
adding rd.auto still works too, if you're lazy. Not sure if there's any reason to keep the bug open...
Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
please double check your grub config and your kernel command line parameter with what dracut suggests: $ sudo dracut --print-cmdline
Happened again on upgrading to Fedora 32 from Fedora 30. [root@i7 ~]# dracut --print-cmdline rd.md.uuid=c5bb489f:38e433db:367adfc7:97def01f root=UUID=c9203359-7154-4ca1-997b-2622b136879b rootfstype=ext4 rootflags=rw,relatime,seclabel,stripe=256 [root@i7 ~]# cat /etc/mdadm.conf ARRAY /dev/md/localhost:root metadata=1.2 name=localhost:root UUID=c5bb489f:38e433db:367adfc7:97def01f But... [root@i7 ~]# grep kernelopts /etc/grub2-efi.cfg set default_kernelopts="root=UUID=c9203359-7154-4ca1-997b-2622b136879b ro quiet rhgb " [root@i7 ~]# grub2-mkconfig 2>/dev/null | egrep "rd.md|kernelopts=" set kernelopts="root=UUID=c9203359-7154-4ca1-997b-2622b136879b ro quiet rhgb " And since kernelopts doesn't tell the kernel to assemble the raid array, none of the BLS entries for specific kernels do either, and my machine doesn't boot.
This message is a reminder that Fedora 32 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora 32 on 2021-05-25. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '32'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 32 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 32 changed to end-of-life (EOL) status on 2021-05-25. Fedora 32 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.