Bug 738035 - Incorrectly assembled array (possibly now corrupted)
Incorrectly assembled array (possibly now corrupted)
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: mdadm (Show other bugs)
16
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Doug Ledford
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-13 13:24 EDT by Andy Burns
Modified: 2011-09-21 16:10 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-09-21 16:10:49 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Andy Burns 2011-09-13 13:24:22 EDT
Description of problem:

I hope this isn't TL;DR but I wanted to be thorough.

I have a machine which was previously booting Centos 5.x
drive layout was as follows ...

/dev/sda 250GB SATA
/dev/sdb 250GB SATA
     sda1 101MB raid partition
     sdb1 101MB raid partition 
          as /dev/md0 (RAID1) for /boot ext3
     sda2 249GB
     sdb2 249GB
          as /dev/md1 (RAID1) for LVM partition 
                member of LVM vgr1
                       multiple LVs inside

/dev/sdc USB CF reader unused
/dev/sdd USB XD reader unused
/dev/sde USB MS reader unused
/dev/sdf USB SD reader unused

/dev/sdg 750GB SATA
/dev/sdh 750GB SATA
/dev/sdi 750GB SATA
/dev/sdj 750GB SATA
/dev/sdk 750GB SATA
/dev/sdl 750GB SATA
/dev/sdm 750GB SATA
/dev/sdn 750GB SATA
     sdg1 750GB raid partition
     sdh1 750GB raid partition
     sdi1 750GB raid partition
     sdj1 750GB raid partition
     sdk1 750GB raid partition
     sdl1 750GB raid partition
     sdk1 750GB raid partition
     sdn1 750GB raid partition
          as /dev/md2 (RAID5) for LVM partition
                member of LVM vgr5
                       multiple LVs inside

All arrays assembled OK during the last boot of Centos on these disks, I still have the /var/log/boot to show that.

I have now installed FC16alpha in this machine.
only /dev/sda and /dev/sdb were selected during anaconda

/dev/md0 and /boot had to be recreated slightly smaller to expand the grub2 embedding area, no problem with this, I used metadata0.90 for compatibility

[   13.139218] md: md0 stopped.
[   13.146291] md: bind<sdb1>
[   13.147681] md: bind<sda1>
[   13.160759] md: raid1 personality registered for level 1
[   13.163943] bio: create slab <bio-1> at 1
[   13.167409] md/raid1:md0: active with 2 out of 2 mirrors
[   13.167759] md0: detected capacity change from 0 to 105775104
[   13.168584] dracut: mdadm: /dev/md0 has been started with 2 drives.

Additionally /dev/md1 assembles OK (though renamed to /dev/md127)

[   13.280733] md: md127 stopped.
[   13.285210] md: bind<sda2>
[   13.286452] md: bind<sdb2>
[   13.302686] md/raid1:md127: active with 2 out of 2 mirrors
[   13.303332] md127: detected capacity change from 0 to 249949716480
[   13.304674] mdadm used greatest stack depth: 3504 bytes left
[   13.305574] dracut: mdadm: /dev/md127 has been started with 2 drives.
[   14.034372] dracut: Scanning devices md127  for LVM volume groups
[   14.104712] dracut: Reading all physical volumes. This may take a while...
[   14.182524] dracut: Found volume group "vgr1" using metadata type lvm2
[   14.538011] dracut: 14 logical volume(s) in volume group "vgr1" now active

There shoudln't be a partition table so this is fine,
the LVs contained in it are good

some "renumbering" of the remaining devices has occured, this shouldn't be a problem, the drive was previously assembling without use of named devices in /etc/mdadm

the devices are now as follows

/dev/sda 250GB SATA
/dev/sdb 250GB SATA
     sda1 100MB raid partition
     sdb1 100MB raid partition 
          as /dev/md0 (RAID1) for /boot ext3
     sda2 249GB
     sdb2 249GB
          as /dev/md1 (RAID1) for LVM partition 
                member of LVM vgr1
                       multiple LVs inside


/dev/sdc 750GB SATA
/dev/sdd USB XD reader unused
/dev/sde USB MS reader unused
/dev/sdf USB SD reader unused
/dev/sdg USB CF reader unused
/dev/sdh 750GB SATA
/dev/sdi 750GB SATA
/dev/sdj 750GB SATA
/dev/sdk 750GB SATA
/dev/sdl 750GB SATA
/dev/sdm 750GB SATA
/dev/sdn 750GB SATA


Somehow mdadm then assembles an array with 5/6 disks :-(


[   25.113299] md: bind<sdh>
[   25.417666] md: bind<sdc>
[   25.635710] md: bind<sdn>
[   25.948352] Adding 2097148k swap on /dev/mapper/vgr1-lvswap.  Priority:0 extents:1 across:2097148k
[   26.049150] md: bind<sdi>
[   26.152274] md: bind<sdj>
[   26.240989] md: bind<sdm1>
[   26.359895] md: bind<sdk1>
[   26.458928] md: bind<sdl>
[   26.512200] xor: automatically using best checksumming function: generic_sse
[   26.517072]    generic_sse:  2772.000 MB/sec
[   26.517077] xor: using function: generic_sse (2772.000 MB/sec)
[   26.546141] raid6: int64x1   2144 MB/s
[   26.563071] raid6: int64x2   2289 MB/s
[   26.580077] raid6: int64x4   2011 MB/s
[   26.597091] raid6: int64x8   1550 MB/s
[   26.614139] raid6: sse2x1    4347 MB/s
[   26.631079] raid6: sse2x2    4457 MB/s
[   26.648080] raid6: sse2x4    6816 MB/s
[   26.648084] raid6: using algorithm sse2x4 (6816 MB/s)
[   26.658548] md: raid6 personality registered for level 6
[   26.658553] md: raid5 personality registered for level 5
[   26.658558] md: raid4 personality registered for level 4
[   26.660510] md/raid:md126: device sdl operational as raid disk 3
[   26.660514] md/raid:md126: device sdj operational as raid disk 2
[   26.660517] md/raid:md126: device sdn operational as raid disk 1
[   26.660520] md/raid:md126: device sdc operational as raid disk 0
[   26.660523] md/raid:md126: device sdh operational as raid disk 4
[   26.666221] md/raid:md126: allocated 6384kB
[   26.667001] md/raid:md126: raid level 5 active with 5 out of 6 devices, algorithm 2


again /dev/md2 has been renamed tro /dev/md126

Whoooah!! This should be an 8 device array :-(

Some disks are being used as raw devices, others still with their partition tables, though sfdisk still thinks the partitions exist on all drives.

This is not the first boot into fedora, so I suppose something previous might have overwritten the partition table and/or metadata, perhaps the 3.0.0 kernel before I upgraded it to 3.1.0-rc5 ?

# sfdisk -l /dev/sd[chijklmn]

Disk /dev/sdc: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty

Disk /dev/sdh: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdh1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdh2          0       -       0          0    0  Empty
/dev/sdh3          0       -       0          0    0  Empty
/dev/sdh4          0       -       0          0    0  Empty

Disk /dev/sdi: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdi1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdi2          0       -       0          0    0  Empty
/dev/sdi3          0       -       0          0    0  Empty
/dev/sdi4          0       -       0          0    0  Empty

Disk /dev/sdj: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdj1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdj2          0       -       0          0    0  Empty
/dev/sdj3          0       -       0          0    0  Empty
/dev/sdj4          0       -       0          0    0  Empty

Disk /dev/sdk: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdk1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdk2          0       -       0          0    0  Empty
/dev/sdk3          0       -       0          0    0  Empty
/dev/sdk4          0       -       0          0    0  Empty

Disk /dev/sdl: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdl1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdl2          0       -       0          0    0  Empty
/dev/sdl3          0       -       0          0    0  Empty
/dev/sdl4          0       -       0          0    0  Empty

Disk /dev/sdm: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdm1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdm2          0       -       0          0    0  Empty
/dev/sdm3          0       -       0          0    0  Empty
/dev/sdm4          0       -       0          0    0  Empty

Disk /dev/sdn: 91201 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdn1   *      0+  91200   91201- 732572001   fd  Linux raid autodetect
/dev/sdn2          0       -       0          0    0  Empty
/dev/sdn3          0       -       0          0    0  Empty
/dev/sdn4          0       -       0          0    0  Empty


I've tried manually assembling with all 8 devices, but that doesn't help, here is what mdadm thinks of the devices, first as raw devices using 
# mdadm --examine /dev/sd[chijklmn]

*********************************************************

/dev/sdc:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6b93 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       32        0      active sync   /dev/sdc

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi
/dev/sdh:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6beb - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8      112        4      active sync   /dev/sdh

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi
/dev/sdi:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6bf9 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     6       8      128        6      spare   /dev/sdi

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi
/dev/sdj:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6c07 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8      144        2      active sync   /dev/sdj

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi
/dev/sdk:
   MBR Magic : aa55
Partition[0] :   1465144002 sectors at           63 (type fd)
/dev/sdl:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6c29 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8      176        3      active sync   /dev/sdl

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi
/dev/sdm:
   MBR Magic : aa55
Partition[0] :   1465144002 sectors at           63 (type fd)
/dev/sdn:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 858169d2:747043a4:9a1b2f67:1e7a0443
  Creation Time : Sun Jun  1 11:59:51 2008
     Raid Level : raid5
  Used Dev Size : 732574464 (698.64 GiB 750.16 GB)
     Array Size : 3662872320 (3493.19 GiB 3750.78 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 2

    Update Time : Tue Sep 13 17:21:15 2011
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : 1e0f6c45 - correct
         Events : 3

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8      208        1      active sync   /dev/sdn

   0     0       8       32        0      active sync   /dev/sdc
   1     1       8      208        1      active sync   /dev/sdn
   2     2       8      144        2      active sync   /dev/sdj
   3     3       8      176        3      active sync   /dev/sdl
   4     4       8      112        4      active sync   /dev/sdh
   5     5       0        0        5      faulty removed
   6     6       8      128        6      spare   /dev/sdi


*********************************************************


Then as partitions (for those that it still thinks exist) using

# mdadm --examine /dev/sd[chijklmn]1

/dev/sdk1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7cf2cd18:33de6f97:35e63bfe:76fc02f2
  Creation Time : Fri Jun 13 16:29:32 2008
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 5128003328 (4890.45 GiB 5251.08 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 2

    Update Time : Mon Sep 12 11:13:14 2011
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
       Checksum : c95946c8 - correct
         Events : 487282

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     7       8      161        7      active sync   /dev/sdk1

   0     0       8      209        0      active sync
   1     1       8      113        1      active sync
   2     2       8       97        2      active sync
   3     3       8      129        3      active sync
   4     4       8      177        4      active sync
   5     5       8      145        5      active sync
   6     6       8      193        6      active sync   /dev/sdm1
   7     7       8      161        7      active sync   /dev/sdk1
/dev/sdm1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7cf2cd18:33de6f97:35e63bfe:76fc02f2
  Creation Time : Fri Jun 13 16:29:32 2008
     Raid Level : raid5
  Used Dev Size : 732571904 (698.64 GiB 750.15 GB)
     Array Size : 5128003328 (4890.45 GiB 5251.08 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 2

    Update Time : Mon Sep 12 11:13:14 2011
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
       Checksum : c95946e6 - correct
         Events : 487282

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     6       8      193        6      active sync   /dev/sdm1

   0     0       8      209        0      active sync
   1     1       8      113        1      active sync
   2     2       8       97        2      active sync
   3     3       8      129        3      active sync
   4     4       8      177        4      active sync
   5     5       8      145        5      active sync
   6     6       8      193        6      active sync   /dev/sdm1
   7     7       8      161        7      active sync   /dev/sdk1


***************************************************

Version-Release number of selected component (if applicable):

kernel.x86_64     3.1.0-0.rc5.git0.0.fc16
mdadm.x86_64      3.2.2-9.fc16
Comment 1 Andy Burns 2011-09-18 16:22:32 EDT
I now think the root cause of this is that udev is not creating the /dev/sd*1 devices for the partitions on all the devices, so have logged that under another bug (BZ#739296)

I am still concerned as to how, when handed 6 raw disks and 2 partitions instead of 8 partitions, mdadm thinks it's doing the correct thing by assembling 5/6 disks into an array and attempting to assemble the two partitions into an array.
Comment 2 Doug Ledford 2011-09-19 12:21:09 EDT
This is a known problem with version 0.90 superblocks.  It was a primary motivating factor in deprecating that superblock version.  They can't tell the difference between a superblock at the end of a whole disk partition and a superblock at the end of a whole disk.  We have moved to version 1.x superblocks which protect against this issue.  I would *strongly* suggest you upgrade the superblock on your arrays by remaking the array using the partitions and not the whole drives, with the exact same options as it was originally created with, except specifying a version 1.0 superblock (which will also place the superblock at the end of the device) and passing the --assume-clean flag to keep it from trying to recreate parity and then seeing if your data is intact.  If it is, then you are good to go (aside from having to update the uuid in the mdadm.conf file to match the new uuid).  If not, then the incorrect assembly has likely caused parity generation to corrupt your array.
Comment 3 Andy Burns 2011-09-19 17:39:55 EDT
That sounds like a reasonable explanation fo what I am seeing, the arrays are several years old so likely to have out of date metadata format.

Although the machine thinks it has assembled the array (sort of!) I hope nothing has been written to the disks, obviously I can't see the LVM PV which was on the array originally, so the LVs within that haven't been mounted.

I don't think parity regeneration has been triggered on the array.

I will try to do as you say once I can presuade the machine to power-on again (that's a completely different issue).

Thank you very much for the comment.
Comment 4 Andy Burns 2011-09-20 16:49:06 EDT
OK, machine is now bootable again (faulty DVB-S2 card removed)

Given that the /dev/sd*1 device files for most of the partitions don't exist, are you saying I should wipe the partition tables, then re-create them (with exactly the same start/end sector obviously) and *then* recreate the array with --assume-clean and --metadata=1.0 

Is metadata1.0 compatible with Centos5.x, if not should I just go with metadata1.2

I assume the order of the partitions within the array is critical? 

I have the dmesg output from the last time the array was successfuly mounted under Centos5.x the device names were slightly different under Centos (ghijklmn instead of chijklmn)

md: Autodetecting RAID arrays.
md: autorun ...
md: considering sdn1 ...
md:  adding sdn1 ...
md:  adding sdm1 ...
md:  adding sdl1 ...
md:  adding sdk1 ...
md:  adding sdj1 ...
md:  adding sdi1 ...
md:  adding sdh1 ...
md:  adding sdg1 ...
md: created md2
md: bind<sdg1>
md: bind<sdh1>
md: bind<sdi1>
md: bind<sdj1>
md: bind<sdk1>
md: bind<sdl1>
md: bind<sdm1>
md: bind<sdn1>
md: running: <sdn1><sdm1><sdl1><sdk1><sdj1><sdi1><sdh1><sdg1>
raid5: device sdn1 operational as raid disk 0
raid5: device sdm1 operational as raid disk 6
raid5: device sdl1 operational as raid disk 4
raid5: device sdk1 operational as raid disk 7
raid5: device sdj1 operational as raid disk 5
raid5: device sdi1 operational as raid disk 3
raid5: device sdh1 operational as raid disk 1
raid5: device sdg1 operational as raid disk 2
raid5: allocated 8462kB for md2
raid5: raid level 5 set md2 active with 8 out of 8 devices, algorithm 2
RAID5 conf printout:
 --- rd:8 wd:8 fd:0
 disk 0, o:1, dev:sdn1
 disk 1, o:1, dev:sdh1
 disk 2, o:1, dev:sdg1
 disk 3, o:1, dev:sdi1
 disk 4, o:1, dev:sdl1
 disk 5, o:1, dev:sdj1
 disk 6, o:1, dev:sdm1
 disk 7, o:1, dev:sdk1
md: ... autorun DONE.
Comment 5 Doug Ledford 2011-09-20 18:12:35 EDT
Yes, you'll need to recreate the partition tables.  And version 1.0 metadata should work with CentOS 5 just fine.  Do *NOT* go with version 1.2 metadata.  That will stick the superblock at the beginning of the partition and will offset the start of your data roughly 1MB into the partition.  When that happens, nothing will be valid any more because your real data will start at the beginning of the partition (and will have been overwritten by the new superblock in addition) but the new array will start reading your real data 1MB into the actual data.  Nothing will be in the right place and the LVM PV will be permanently hosed.

As for the order of the partitions, it is critical.  However, as long as you use the --assume-clean flag, you can remake the array as many times as you need to get the order right.  So, first try to get the order right by using the dmesg from the last successful boot.  When it says:

raid5: device sdnl operational as raid disk 0

that means the devices sdnl that existed before should be the first disk you list on the list of devices to create the array from.  Number 1 is next, and so on.  So, from what you listed above, the create command would be something like this:

mdadm -C /dev/md2 -l5 -n8 -e1.0 --name=md2 --chunk=64 --assume-clean /dev/sd[nhgiljmk]1

However, I note that you asked if version 1.0 metadata is ok with CentOS 5.  In CentOS we have an older version of mdadm (2.6.9) and although I know it supports --assume-clean, I'm not positive how well version 1.0 metadata is supported.  You'll just have to try it and see how it goes.
Comment 6 Andy Burns 2011-09-21 16:10:49 EDT
This issue is now solved for me, the problem was that 6 of the 8 disks had previously been members of a different array using md-on-rawdisk, then 2 additional disks were purchased and 8 partitions created, with an md-on-partitions array (used for several years under CentOS5) 

My issue only was noticed when changing to FC16, where udev apparently sees the rawdisk superblock first and uses it, so simply zeroing the superblock on the raw device allowed the metadata on the partition to be "unshadowed" and was used correctly on the next reboot, no data lost, I will probably still take the suggestion to to upgrade to metadata1.0 though.

Thanks.

Note You need to log in before you can comment on or make changes to this bug.