Bug 787832

Jes, we evidently need to coordinate with the systemd people on moving the md-device-map file from /dev/md to /run/mdadm.  I didn't realize they had any hooks on that file in systemd.

Of course, that line is also supposed to be a failsafe, there is also the question of why the array didn't simply come up during the udev device scans.  We really need answers to both of those questions I think.

Thomas, can you please verify that you have a /lib/udev/rules.d/65-md-incremental.rules file on your system.  Then can you please post the output of fdisk -l /dev/sdb here as well as the output of blkid -o udev /dev/sdb, and if you have any partitions on /dev/sdb, then blkid -o udev /dev/sdb? as well.

That should get us started trying to figure out why your system didn't boot up properly.

Jes, since an update is possibly rendering machines unbootable, this is going to need to be a very high priority and we likely should file a release-engineering ticket to remove the existing update from the repos and push a new update ASAP.

In fact Jes, I think we need to move the location of the md-device-map file back to /dev/md/ for f15 and f16.  I think that change is too risky for either of those releases.  The reason, which just came to mind, is that if people need early boot md work, and they don't rebuild all their existing initramfs images, then the old mdadm on the initramfs and the new mdadm on the root filesystem won't agree about where the md-device-map file should be and we could get races and other bad things happening.

Doug,

Was just checking this before going to bed.

mdadm-3.2.3-3 does *not* have the move of /dev/md/md-device-map, that is
only in mdadm-3.2.3-5 which I haven't pushed upstream yet.

Why there is no /dev/md/md-device-map showing up on Thomas' system will
be due to something else.

Thomas, did you run dracut -f "" after updating the mdadm package?
Otherwise you will end up with a different version of mdadm/mdmon in
the initramfs compared to what is on the root file system.

Jes

Thomas,

Could you please provide the output of /proc/mdstat and
mdadm --detail /dev/md<X> of the raid from the case where it booted
correctly.

Thanks,
Jes

Jes, whew, that's good to hear ;-)

Yes there is a udev rules file:
# ll /lib/udev/rules.d/65-md-incremental.rules
-rw-r--r-- 1 root root 3348 Nov 22 11:32
/lib/udev/rules.d/65-md-incremental.rules

Here's the output of fdisk -l for both raid1 disks:

# fdisk -l /dev/sdb

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7cf498aa

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *          63      409662      204800   83  Linux
/dev/sdb2          409663  1953520064   976555201   fd  Linux raid autodetect


# fdisk -l /dev/sdc

Disk /dev/sdc: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0005be44

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63      409662      204800   83  Linux
/dev/sdc2          409663  1953520064   976555201   fd  Linux raid autodetect

On sdb and sdc there is an old boot partition. I had these two disks only a
longer time before. After adding sda as boot disk I setup fc15 from scratch and
used the sdb/sdc as raid1 for my data. The os resides on sda completely.


The blkid output:

# blkid -o udev /dev/sdb
no output

blkid for the partitions:
# blkid -o udev /dev/sdb1
ID_FS_UUID=5d2bece0-e614-46bc-8bae-bad30faa6795
ID_FS_UUID_ENC=5d2bece0-e614-46bc-8bae-bad30faa6795
ID_FS_SEC_TYPE=ext2
ID_FS_TYPE=ext3

# blkid -o udev /dev/sdb2
ID_FS_UUID=5cf496a1-e554-585d-bfe7-8010bc810f04
ID_FS_UUID_ENC=5cf496a1-e554-585d-bfe7-8010bc810f04
ID_FS_TYPE=linux_raid_member

# blkid -o udev /dev/sdc1
ID_FS_UUID=a403ca3a-4f43-4986-8c8d-aee3f6d3564d
ID_FS_UUID_ENC=a403ca3a-4f43-4986-8c8d-aee3f6d3564d
ID_FS_SEC_TYPE=ext2
ID_FS_TYPE=ext3

# blkid -o udev /dev/sdc2
ID_FS_UUID=5cf496a1-e554-585d-bfe7-8010bc810f04
ID_FS_UUID_ENC=5cf496a1-e554-585d-bfe7-8010bc810f04
ID_FS_TYPE=linux_raid_member


Doug, during my investigations I tried the /sbin/mdadm -IRs on the maintenance
console. This command didn't assemble the raid1. A mdadm --auto-detect does the
job. As far as I know the options -IRs mean incremental, run, scan. Why the
scan didn't work out and the --auto-detect works? Maybe that helps.


Jes, I didn't run a dracut -f after updating mdadm. It was included in a yum
update. I searched for the changes after having problems during boot and
noticed a change in mdadm. I went back to the prior version and everything
works again.

The answers to your questions:

# cat /proc/mdstat 
Personalities : [raid1] 
md127 : active raid1 sdb2[0] sdc2[1]
      976555136 blocks [2/2] [UU]

unused devices: <none>


# mdadm --detail /dev/md127
/dev/md127:
        Version : 0.90
  Creation Time : Fri Oct 23 17:51:12 2009
     Raid Level : raid1
     Array Size : 976555136 (931.32 GiB 999.99 GB)
  Used Dev Size : 976555136 (931.32 GiB 999.99 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127
    Persistence : Superblock is persistent

    Update Time : Mon Feb  6 23:13:41 2012
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 5cf496a1:e554585d:bfe78010:bc810f04
         Events : 0.1963

    Number   Major   Minor   RaidDevice State
       0       8       18        0      active sync   /dev/sdb2
       1       8       34        1      active sync   /dev/sdc2

Thomas,

We think we know what might be wrong. Can you provide a copy of your
/etc/mdadm.conf

Thanks,
Jes

Created attachment 559780 [details]
/etc/mdadm.conf file

requested mdadm.conf file

Thomas, we have it figured out.  Your system is configured not to boot.  We think that the recent mdadm update might have started honoring something that the old mdadm was not, and hence the change.

However, here's the deal.  You have an old version 0.90 superblock array.  In the mdadm.conf file, the line AUTO +imsm +1.x -all tells mdadm -I invocations to start any imsm or version 1.x superblock arrays, but not ddf or version 0.90 arrays.  Now, if you had an ARRAY line in the mdadm.conf file with your array's uuid, then it would override the fact that the conf file is telling mdadm -I not to assemble version 0.90 arrays and it would assemble your version 0.90 array.

The fact that mdadm --auto-detect works is because you are using version 0.90 arrays, and the kernel will start those itself.  But, that support is going away in the future, so best to have your array getting started by mdadm properly, not via the kernel.

Actually, the big question in my mind, and the one we'll work on here, is why did your system *ever* work.

So, in order to get your system booting properly again, there are three options (listed in order of preference):

1) Recreate your array with a version 1.0 superblock (do not use 1.1 or 1.2, those superblocks would not occupy the same space on disk as the version 0.90 superblock does and would in fact overwrite some of your data...to switch to one of them means you must do a backup and restore of your data).  It can be done over your existing superblocks without damaging your data (raid1 is easy to do this with, other raids you can too, but you have to make sure and match things like chunk sizes and such).  You should use the size option if you do this just to make sure that the new superblock doesn't try to utilize more space than the 0.90 superblock (aka, check the device's current size, that of the md raid device, and when you recreate the raid1 array, pass --size= and make it match, that will force mdadm to use exactly the same number of data blocks on the new array as on the old).  You can also pass --assume-clean to avoid a resync.  Once this is done, your array should start coming up automatically on each boot by virtue of it being a version 1.x array.  If you do this you should also change the partition type for the raid partitions to type 0xfd non-fs data.  The old raid autodetect partition type is being deprecated raid along with autodetection.  Something like the following should do the trick:

mdadm -S /dev/md127
fdisk /dev/sd[bc] (change type to 0xfd)
mdadm -C /dev/md/home (or whatever you want to call it, version 1.x arrays are named instead of numbered, so you would also want to update fstab to use the array by name and not number) -e1.0 --size=976555136 -l1 -n2 --assume-clean /dev/sd[bc]2

2) Add an ARRAY line to mdadm.conf for your existing array (and you might want to do this step even if you also do the first option).

mdadm -Db /dev/md127 >> /etc/mdadm.conf will do the trick

3) Enable auto assembly of unknown version .90 arrays by changing the AUTO line in the mdadm.conf file to read AUTO +all -ddf or AUTO +imsm +1.x +0.90 -all

Any of those options should solve your problem.  In the mean time, we need to figure out why your system went from working to not-working and make sure that this isn't going to bite more people than we think it is.

Doug, thanx for the analysis and the options fixing the problem. I think 3) is the choise I will use. I've running some jobs at the moment and will test the solution with the current mdadm package after the jobs completed. I will update the bug with my results.

I checked an old config backup of my server (backup before installing Fedora 15 on the server). The /etc/mdadm.conf contained the same lines as today. 

A little bit of history which perhaps help to figure out if there maybe are more people having the same problem.

The raid1 array I created with a Fedora 13 release. I had only two disks in my server wich are forming the raid. The raid1 was created by the Ferdora installer.

A couple of month ago I added a new ssd disk to my server and made this disk the boot disk having the os on it. The raid1 I used for my data only from this time on. During install of Fedora 15 the raid1 was not touched by the installer. But the raid1 was detected automatically until the last update of mdadm.

Doug, another question. Is there an explanation why yum downgrade mdadm results in the version 3.1.5 of mdadm? I did not find the previous version mdadm-3.2.2-15.fc15.x86_64 in the updates repository. I had to search the net for this particular package and install it locally after downloading it.

Doug, here are the results of my tests. Adding +0.90 in the mdadm.conf fixes the problem for me. My system boots now with the current version of mdadm.

Here's a suggestion to solve the problem:
in the post section of the package check if there are existing raid arrays on the system. Check the superblock of each array. From the superblock versions you can decide how to change the mdadm.conf to work with the current available raid arrays. My suggestion assumes that mdadm will be backward compatible and able to work with older superblock versions in the future.

Hi Thomas,

Glad to hear it is indeed working for you with the update. Detecting arrays
in the post install script and then modifying people's mdadm.conf is risky
as some things could have been put in there on purpose.

I suspect the best thing we can do here is to provide documentation, and
possible emit a warning in the post install script if a 0.90 array without
a line is detected.

Cheers,
Jes

Thomas,

I have determined the exact culprit for the problems you were seeing.
Basically mdadm-3.2.2 had a bug where any auto information in mdadm.conf
was not respected. This was fixed by the commit below, which is included
in the mdadm-3.2.3 update that I pushed into Fedora 15 and Fedora 16.

Given that the actual bug was that mdadm didn't behave as documented, and
it does so now, I am reluctant to try and make changes to it, but rather
make a note about this in the documentation.

Cheers,
Jes

commit b451aa4846c5ccca5447a6b6d45e5623b8c8e961
Author: NeilBrown <neilb>
Date:   Thu Oct 6 13:00:28 2011 +1100

    Fix handling for "auto" line in mdadm.conf
    
    Two problems.
    
    1/ pol_merge was ignoring the pol_auto tag so any 'auto' information
       was lost
    2/ If a device had not path (e.g. loop devices) or if there were no
       path-based policies, we didn't bother looking for policy at all.
       So path-independant policies were ignored.
    
    Reported-by: Christian Boltz <suse-beta>
    Signed-off-by: NeilBrown <neilb>