Hide Forgot
Description of problem: I haven't kept entirely abreast of changes in fedora in recent versions, so I hope udev is the correct component, rather than it being devicekit/hal/systemd/randomvoodoo that is responsible. I have 10 SATA disks in my machine 2x250GB used for 2 RAID1 partitions md0=sda1+sdb1 md1=sda2+sdb2 8x750GB used for 1 RAID5 partition md2=sdc1+sdh1+sdi1+sdj1+sdk1+sdl1+sdm1+sdn1 sdd/sde/sdf/sdg are irrelevant as they are a USB memory card reader Version-Release number of selected component (if applicable): libudev.x86_64 173-3.fc16 systemd.x86_64 35-1.fc16 udev.x86_64 173-3.fc16 mdadm.x86_64 3.2.2-9.fc16 How reproducible: 100% Steps to Reproduce: 1. boot system 2. ls /dev/sd[chijklmn]1 3. cat /proc/mdstat Actual results: /dev/sd[km]1 devices are present /dev/sd[chijln]1 devices are absent crying sounds due to lost 5TB array Expected results: /dev/sd[chijklmn]1 partitions should all exist all raid arrays detected and assembled correctly Additional info: From dmesg and fdisk it is clear the partitions do actually exist on all disks, but the devices for them have not been created, as seen from systemctl Why mdadm has chosen to assemble 6/8ths of my array from the raw disks rather than the partitions is a separate mystery. # dmesg | grep sd[abchijklmn] [ 2.462931] sd 0:0:0:0: [sda] 488397168 512-byte logical blocks: (250 GB/232 GiB) [ 2.464314] sd 0:0:0:0: [sda] Write Protect is off [ 2.464319] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 [ 2.464841] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.494755] sda: sda1 sda2 [ 2.499699] sd 0:0:0:0: [sda] Attached SCSI disk [ 2.929892] sd 1:0:0:0: [sdb] 488397168 512-byte logical blocks: (250 GB/232 GiB) [ 2.931357] sd 1:0:0:0: [sdb] Write Protect is off [ 2.931362] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 [ 2.931821] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 2.945278] sdb: sdb1 sdb2 [ 2.949733] sd 1:0:0:0: [sdb] Attached SCSI disk [ 8.367124] sd 9:0:0:0: [sdc] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 8.368310] sd 9:0:0:0: [sdc] Write Protect is off [ 8.368314] sd 9:0:0:0: [sdc] Mode Sense: 00 3a 00 00 [ 8.368820] sd 9:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 8.390877] sdc: sdc1 [ 8.394656] sd 9:0:0:0: [sdc] Attached SCSI disk [ 8.870890] sd 10:0:0:0: [sdh] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 8.872125] sd 10:0:0:0: [sdh] Write Protect is off [ 8.872129] sd 10:0:0:0: [sdh] Mode Sense: 00 3a 00 00 [ 8.872631] sd 10:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 8.898684] sdh: sdh1 [ 8.902489] sd 10:0:0:0: [sdh] Attached SCSI disk [ 9.371637] sd 11:0:0:0: [sdi] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 9.373172] sd 11:0:0:0: [sdi] Write Protect is off [ 9.373177] sd 11:0:0:0: [sdi] Mode Sense: 00 3a 00 00 [ 9.373590] sd 11:0:0:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 9.397791] sdi: sdi1 [ 9.401482] sd 11:0:0:0: [sdi] Attached SCSI disk [ 9.871958] sd 12:0:0:0: [sdj] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 9.873457] sd 12:0:0:0: [sdj] Write Protect is off [ 9.873462] sd 12:0:0:0: [sdj] Mode Sense: 00 3a 00 00 [ 9.873920] sd 12:0:0:0: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 9.895867] sdj: sdj1 [ 9.899565] sd 12:0:0:0: [sdj] Attached SCSI disk [ 10.374225] sd 13:0:0:0: [sdk] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 10.375322] sd 13:0:0:0: [sdk] Write Protect is off [ 10.375326] sd 13:0:0:0: [sdk] Mode Sense: 00 3a 00 00 [ 10.375739] sd 13:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.399210] sdk: sdk1 [ 10.403270] sd 13:0:0:0: [sdk] Attached SCSI disk [ 10.874010] sd 14:0:0:0: [sdl] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 10.875879] sd 14:0:0:0: [sdl] Write Protect is off [ 10.875883] sd 14:0:0:0: [sdl] Mode Sense: 00 3a 00 00 [ 10.876339] sd 14:0:0:0: [sdl] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 10.897014] sdl: sdl1 [ 10.900709] sd 14:0:0:0: [sdl] Attached SCSI disk [ 11.375219] sd 15:0:0:0: [sdm] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 11.376258] sd 15:0:0:0: [sdm] Write Protect is off [ 11.376262] sd 15:0:0:0: [sdm] Mode Sense: 00 3a 00 00 [ 11.376676] sd 15:0:0:0: [sdm] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 11.401912] sdm: sdm1 [ 11.405670] sd 15:0:0:0: [sdm] Attached SCSI disk [ 11.875884] sd 16:0:0:0: [sdn] 1465149168 512-byte logical blocks: (750 GB/698 GiB) [ 11.877069] sd 16:0:0:0: [sdn] Write Protect is off [ 11.877073] sd 16:0:0:0: [sdn] Mode Sense: 00 3a 00 00 [ 11.877558] sd 16:0:0:0: [sdn] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 11.882893] sdn: sdn1 [ 11.886610] sd 16:0:0:0: [sdn] Attached SCSI disk ============================================================ # sfdisk -l /dev/sd[abchijklmn] Disk /dev/sda: 30401 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sda1 * 0+ 12 13- 103398+ fd Linux raid autodetect /dev/sda2 13 30400 30388 244091610 fd Linux raid autodetect /dev/sda3 0 - 0 0 0 Empty /dev/sda4 0 - 0 0 0 Empty Disk /dev/sdb: 30401 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdb1 * 0+ 12 13- 103398+ fd Linux raid autodetect /dev/sdb2 13 30400 30388 244091610 fd Linux raid autodetect /dev/sdb3 0 - 0 0 0 Empty /dev/sdb4 0 - 0 0 0 Empty Disk /dev/sdc: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdc1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdc2 0 - 0 0 0 Empty /dev/sdc3 0 - 0 0 0 Empty /dev/sdc4 0 - 0 0 0 Empty Disk /dev/sdh: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdh1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdh2 0 - 0 0 0 Empty /dev/sdh3 0 - 0 0 0 Empty /dev/sdh4 0 - 0 0 0 Empty Disk /dev/sdi: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdi1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdi2 0 - 0 0 0 Empty /dev/sdi3 0 - 0 0 0 Empty /dev/sdi4 0 - 0 0 0 Empty Disk /dev/sdj: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdj1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdj2 0 - 0 0 0 Empty /dev/sdj3 0 - 0 0 0 Empty /dev/sdj4 0 - 0 0 0 Empty Disk /dev/sdk: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdk1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdk2 0 - 0 0 0 Empty /dev/sdk3 0 - 0 0 0 Empty /dev/sdk4 0 - 0 0 0 Empty Disk /dev/sdl: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdl1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdl2 0 - 0 0 0 Empty /dev/sdl3 0 - 0 0 0 Empty /dev/sdl4 0 - 0 0 0 Empty Disk /dev/sdm: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdm1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdm2 0 - 0 0 0 Empty /dev/sdm3 0 - 0 0 0 Empty /dev/sdm4 0 - 0 0 0 Empty Disk /dev/sdn: 91201 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System /dev/sdn1 * 0+ 91200 91201- 732572001 fd Linux raid autodetect /dev/sdn2 0 - 0 0 0 Empty /dev/sdn3 0 - 0 0 0 Empty /dev/sdn4 0 - 0 0 0 Empty =============================================================== # systemctl --full | grep sd[abchijklmn] sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host10-target10:0:0-10:0:0:0-block-sdh.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host11-target11:0:0-11:0:0:0-block-sdi.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host12-target12:0:0-12:0:0:0-block-sdj.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host13-target13:0:0-13:0:0:0-block-sdk-sdk1.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host13-target13:0:0-13:0:0:0-block-sdk.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host14-target14:0:0-14:0:0:0-block-sdl.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host15-target15:0:0-15:0:0:0-block-sdm-sdm1.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host15-target15:0:0-15:0:0:0-block-sdm.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host16-target16:0:0-16:0:0:0-block-sdn.device loaded active plugged sys-devices-pci0000:00-0000:00:1c.0-0000:05:00.0-0000:07:01.0-host9-target9:0:0-9:0:0:0-block-sdc.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host0-target0:0:0-0:0:0:0-block-sda-sda1.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host0-target0:0:0-0:0:0:0-block-sda-sda2.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host0-target0:0:0-0:0:0:0-block-sda.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host1-target1:0:0-1:0:0:0-block-sdb-sdb1.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host1-target1:0:0-1:0:0:0-block-sdb-sdb2.device loaded active plugged sys-devices-pci0000:00-0000:00:1f.2-host1-target1:0:0-1:0:0:0-block-sdb.device loaded active plugged ===================================================== # dmesg | grep sd[abchijklmn] [ 12.998171] md: bind<sdb1> [ 12.999539] md: bind<sda1> [ 13.151445] md: bind<sda2> [ 13.154974] md: bind<sdb2> [ 22.277951] md: bind<sdk1> [ 22.458953] md: bind<sdm1> [ 22.557396] md: bind<sdc> [ 22.627100] md: bind<sdh> [ 22.693853] md: bind<sdi> [ 22.808620] md: bind<sdn> [ 22.886248] md: bind<sdl> [ 23.032700] md: bind<sdj> [ 23.245569] md/raid:md125: device sdj operational as raid disk 2 [ 23.245573] md/raid:md125: device sdl operational as raid disk 3 [ 23.245577] md/raid:md125: device sdn operational as raid disk 1 [ 23.245580] md/raid:md125: device sdh operational as raid disk 4 [ 23.245583] md/raid:md125: device sdc operational as raid disk 0 [ 23.252059] disk 0, o:1, dev:sdc [ 23.252062] disk 1, o:1, dev:sdn [ 23.252065] disk 2, o:1, dev:sdj [ 23.252067] disk 3, o:1, dev:sdl [ 23.252070] disk 4, o:1, dev:sdh [ 27.880532] md/raid:md126: device sdm1 operational as raid disk 6 [ 27.880536] md/raid:md126: device sdk1 operational as raid disk 7 [ 27.887489] disk 6, o:1, dev:sdm1 [ 27.887492] disk 7, o:1, dev:sdk1 [ 1893.479288] md: unbind<sdj> [ 1893.486483] md: export_rdev(sdj) [ 1893.488009] md: unbind<sdl> [ 1893.489670] md: export_rdev(sdl) [ 1893.490662] md: unbind<sdn> [ 1893.492560] md: export_rdev(sdn) [ 1893.493543] md: unbind<sdi> [ 1893.495444] md: export_rdev(sdi) [ 1893.496523] md: unbind<sdh> [ 1893.512221] md: export_rdev(sdh) [ 1893.512827] md: unbind<sdc> [ 1893.520471] md: export_rdev(sdc) [ 1912.703459] md: unbind<sdm1> [ 1912.713216] md: export_rdev(sdm1) [ 1912.714640] md: unbind<sdk1> [ 1912.715258] md: export_rdev(sdk1)
Attempting to manually force partitions to be created does not seem to help [root@xen ~]# kpartx -av /dev/sdc add map sdc1 (253:14): 0 1465144002 linear /dev/sdc 63 [root@xen ~]# kpartx -av /dev/sdh add map sdh1 (253:15): 0 1465144002 linear /dev/sdh 63 [root@xen ~]# kpartx -av /dev/sdi add map sdi1 (253:16): 0 1465144002 linear /dev/sdi 63 [root@xen ~]# kpartx -av /dev/sdj add map sdj1 (253:17): 0 1465144002 linear /dev/sdj 63 [root@xen ~]# kpartx -av /dev/sdk add map sdk1 (253:18): 0 1465144002 linear /dev/sdk 63 [root@xen ~]# kpartx -av /dev/sdl add map sdl1 (253:19): 0 1465144002 linear /dev/sdl 63 [root@xen ~]# kpartx -av /dev/sdm add map sdm1 (253:20): 0 1465144002 linear /dev/sdm 63 [root@xen ~]# kpartx -av /dev/sdn add map sdn1 (253:21): 0 1465144002 linear /dev/sdn 63 # ls /dev/sd[a-n]1 /dev/sda1 /dev/sdb1 /dev/sdk1 /dev/sdm1 the sdc1/sdh1/sdj1/sdl1/sdn1 partitions are still missing
If dracut detects raid signatures on the bare drive, it will remove the partitions. What is the output of: # udevadm info --query=all --name=/dev/sdc
(In reply to comment #2) > If dracut detects raid signatures on the bare drive, it will remove the > partitions > > What is the output of: > > # udevadm info --query=all --name=/dev/sdc The machine is "reluctant" to power-on at the moment, it may take me a day or two to get it booted and paste results ... thanks ...
OK, machine booting again (removed faulty DVB-S2 card) # udevadm info --query=all --name=/dev/sdc P: /devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:07:01.0/host9/target9:0:0/9:0:0:0/block/sdc N: sdc S: disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1EQ317024 S: disk/by-id/scsi-SATA_SAMSUNG_HD753LJS13UJ1EQ317024 S: disk/by-path/pci-0000:07:01.0-scsi-0:0:0:0 S: disk/by-id/wwn-0x50000f000b130742 E: UDEV_LOG=3 E: DEVPATH=/devices/pci0000:00/0000:00:1c.0/0000:05:00.0/0000:07:01.0/host9/target9:0:0/9:0:0:0/block/sdc E: MAJOR=8 E: MINOR=32 E: DEVNAME=/dev/sdc E: DEVTYPE=disk E: SUBSYSTEM=block E: ID_ATA=1 E: ID_TYPE=disk E: ID_BUS=ata E: ID_MODEL=SAMSUNG_HD753LJ E: ID_MODEL_ENC=SAMSUNG\x20HD753LJ\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20 E: ID_REVISION=1AA01190 E: ID_SERIAL=SAMSUNG_HD753LJ_S13UJ1EQ317024 E: ID_SERIAL_SHORT=S13UJ1EQ317024 E: ID_ATA_WRITE_CACHE=1 E: ID_ATA_WRITE_CACHE_ENABLED=1 E: ID_ATA_FEATURE_SET_HPA=1 E: ID_ATA_FEATURE_SET_HPA_ENABLED=1 E: ID_ATA_FEATURE_SET_PM=1 E: ID_ATA_FEATURE_SET_PM_ENABLED=1 E: ID_ATA_FEATURE_SET_SECURITY=1 E: ID_ATA_FEATURE_SET_SECURITY_ENABLED=0 E: ID_ATA_FEATURE_SET_SECURITY_ERASE_UNIT_MIN=168 E: ID_ATA_FEATURE_SET_SECURITY_ENHANCED_ERASE_UNIT_MIN=168 E: ID_ATA_FEATURE_SET_SMART=1 E: ID_ATA_FEATURE_SET_SMART_ENABLED=1 E: ID_ATA_FEATURE_SET_AAM=1 E: ID_ATA_FEATURE_SET_AAM_ENABLED=0 E: ID_ATA_FEATURE_SET_AAM_VENDOR_RECOMMENDED_VALUE=254 E: ID_ATA_FEATURE_SET_AAM_CURRENT_VALUE=0 E: ID_ATA_FEATURE_SET_PUIS=1 E: ID_ATA_FEATURE_SET_PUIS_ENABLED=0 E: ID_ATA_FEATURE_SET_APM=1 E: ID_ATA_FEATURE_SET_APM_ENABLED=0 E: ID_ATA_DOWNLOAD_MICROCODE=1 E: ID_ATA_SATA=1 E: ID_ATA_SATA_SIGNAL_RATE_GEN2=1 E: ID_ATA_SATA_SIGNAL_RATE_GEN1=1 E: ID_WWN=0x50000f000b130742 E: ID_WWN_WITH_EXTENSION=0x50000f000b130742 E: ID_SCSI_COMPAT=SATA_SAMSUNG_HD753LJS13UJ1EQ317024 E: ID_PATH=pci-0000:07:01.0-scsi-0:0:0:0 E: ID_PATH_TAG=pci-0000_07_01_0-scsi-0_0_0_0 E: ID_FS_VERSION=0.90.0 E: ID_FS_UUID=858169d2-7470-43a4-9a1b-2f671e7a0443 E: ID_FS_UUID_ENC=858169d2-7470-43a4-9a1b-2f671e7a0443 E: ID_FS_TYPE=linux_raid_member E: ID_FS_USAGE=raid E: UDISKS_PRESENTATION_NOPOLICY=0 E: MD_LEVEL=raid5 E: MD_DEVICES=6 E: MD_UUID=858169d2:747043a4:9a1b2f67:1e7a0443 E: MD_UPDATE_TIME=1212317991 E: MD_EVENTS=3 E: UDISKS_ATA_SMART_IS_AVAILABLE=1 E: DEVLINKS=/dev/disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1EQ317024 /dev/disk/by-id/scsi-SATA_SAMSUNG_HD753LJS13UJ1EQ317024 /dev/disk/by-path/pci-0000:07:01.0-scsi-0:0:0:0 /dev/disk/by-id/wwn-0x50000f000b130742 E: TAGS=:systemd:
(In reply to comment #4) > OK, machine booting again (removed faulty DVB-S2 card) > > # udevadm info --query=all --name=/dev/sdc > .... > E: ID_FS_TYPE=linux_raid_member > E: ID_FS_USAGE=raid Ok... you clearly have an ?old? raid signature on /dev/sdc. You should get rid of it. something like: # mdadm --zero-superblock /dev/sdc should do the job. You might want to check the other drives, too.
OK, thinking about the history of these disks, many years ago I did create an array of 6 disks on them (I couldn't honestly remember at this point whether this would be on the raw disks, or on partitions) I then purchased 2 additional disks and a separate 8-channel PCIX SATA controller, attached all 8 disks to it, and recreated it as an 8 disk array (definitely using partitions). Are you saying it's feasible that sufficient evidence (the original metadata) remains from the former 6 raw disk array, that udev sees it first and takes it as gospel, ignoring the later 8 partition based array? If, so this starts to fit with advice I've received on the "see also" BZ#730835, to delete and recreate the missing partitions, recreate the array with metadata as v1.0 instead of v0.9, and use --assume-clean to prevent mdadm regenerating parity. Keeping fingers crossed that the contents are still there ... Thanks.
OK, I did mdadm --zero-superblock on each of the 6 drives which previously had md-on-rawdisks, rebooted and mdadm correctly assembled the 8-member array using md-on-partitions, the LVM contents look happy. so the old superblock on the raw drive was "masking" the newer one in the partition. I think I will take the suggestion elewhere to upgrade to metadata1.0 though. Thanks very much :-)