Hide Forgot
Description of problem: I'm seeing this issue while running raid image failure testing with allocation turned on. This may be related to bug 801571. Scenario kill_random_synced_raid5_3legs: Kill random leg of synced 3 leg raid5 volume(s) ********* RAID hash info for this scenario ********* * names: synced_random_raid5_3legs_1 * sync: 1 * type: raid5 * -m |-i value: 3 * leg devices: /dev/sdg1 /dev/sdf1 /dev/sde1 /dev/sdc1 * failpv(s): /dev/sdc1 * failnode(s): taft-01 * raid fault policy: allocate ****************************************************** Creating raids(s) on taft-01... taft-01: lvcreate --type raid5 -i 3 -n synced_random_raid5_3legs_1 -L 500M black_bird /dev/sdg1:0-1000 /dev/sdf1:0-1000 /dev/sde1:0-1000 /dev/sdc1:0-1000 RAID Structure(s): LV Attr LSize Copy% Devices synced_random_raid5_3legs_1 rwi-a-r- 504.00m synced_random_raid5_3legs_1_rimage_0(0),synced_random_raid5_3legs_1_rimage_1(0),synced_random_raid5_3legs_1_rimage_2(0),synced_random_raid5_3legs_1_rimage_3(0) [synced_random_raid5_3legs_1_rimage_0] Iwi-aor- 168.00m /dev/sdg1(1) [synced_random_raid5_3legs_1_rimage_1] Iwi-aor- 168.00m /dev/sdf1(1) [synced_random_raid5_3legs_1_rimage_2] Iwi-aor- 168.00m /dev/sde1(1) [synced_random_raid5_3legs_1_rimage_3] Iwi-aor- 168.00m /dev/sdc1(1) [synced_random_raid5_3legs_1_rmeta_0] ewi-aor- 4.00m /dev/sdg1(0) [synced_random_raid5_3legs_1_rmeta_1] ewi-aor- 4.00m /dev/sdf1(0) [synced_random_raid5_3legs_1_rmeta_2] ewi-aor- 4.00m /dev/sde1(0) [synced_random_raid5_3legs_1_rmeta_3] ewi-aor- 4.00m /dev/sdc1(0) PVS IN VG: /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 PV=/dev/sdc1 synced_random_raid5_3legs_1_rimage_3: 1 synced_random_raid5_3legs_1_rmeta_3: 1 Creating ext on top of mirror(s) on taft-01... mke2fs 1.41.12 (17-May-2010) Mounting mirrored ext filesystems on taft-01... Writing verification files (checkit) to mirror(s) on... ---- taft-01 ---- Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure Verifying files (checkit) on mirror(s) on... ---- taft-01 ---- Disabling device sdc on taft-01 Attempting I/O to cause mirror down conversion(s) on taft-01 10+0 records in 10+0 records out 41943040 bytes (42 MB) copied, 0.536352 s, 78.2 MB/s Verifying current sanity of lvm after the failure RAID Structure(s): /dev/sdc1: read failed after 0 of 512 at 145669554176: Input/output error /dev/sdc1: read failed after 0 of 512 at 145669664768: Input/output error /dev/sdc1: read failed after 0 of 512 at 0: Input/output error /dev/sdc1: read failed after 0 of 512 at 4096: Input/output error /dev/sdc1: read failed after 0 of 2048 at 0: Input/output error Couldn't find device with uuid LOdwxl-cgjB-XCHH-e4j0-FNry-aZlG-D3Edvk. LV Attr LSize Copy% Devices synced_random_raid5_3legs_1 rwi-aor- 504.00m synced_random_raid5_3legs_1_rimage_0(0),synced_random_raid5_3legs_1_rimage_1(0),synced_random_raid5_3legs_1_rimage_2(0),synced_random_raid5_3legs_1_rimage_3(0) [synced_random_raid5_3legs_1_rimage_0] iwi-aor- 168.00m /dev/sdg1(1) [synced_random_raid5_3legs_1_rimage_1] iwi-aor- 168.00m /dev/sdf1(1) [synced_random_raid5_3legs_1_rimage_2] iwi-aor- 168.00m /dev/sde1(1) [synced_random_raid5_3legs_1_rimage_3] iwi-aor- 168.00m unknown device(1) [synced_random_raid5_3legs_1_rmeta_0] ewi-aor- 4.00m /dev/sdg1(0) [synced_random_raid5_3legs_1_rmeta_1] ewi-aor- 4.00m /dev/sdf1(0) [synced_random_raid5_3legs_1_rmeta_2] ewi-aor- 4.00m /dev/sde1(0) [synced_random_raid5_3legs_1_rmeta_3] ewi-aor- 4.00m unknown device(0) Verifying FAILED device /dev/sdc1 is *NOT* in the volume(s) Verifying IMAGE device /dev/sdg1 *IS* in the volume(s) Verifying IMAGE device /dev/sdf1 *IS* in the volume(s) Verifying IMAGE device /dev/sde1 *IS* in the volume(s) verify the rimage/rmeta dm devices remain after the failures Checking EXISTENCE and STATE of synced_random_raid5_3legs_1_rimage_3 on: taft-01there should not be an 'unknown' device associated with synced_random_raid5_3legs_1_rimage_3 on taft-01 Mar 8 17:51:20 taft-01 kernel: device-mapper: raid: Reshaping arrays not yet supported. Mar 8 17:51:20 taft-01 kernel: device-mapper: table: 253:11: raid: Unable to assemble array: Invalid superblocks Mar 8 17:51:20 taft-01 kernel: device-mapper: ioctl: error adding target to table Mar 8 17:51:20 taft-01 lvm[3262]: device-mapper: reload ioctl on failed: Invalid argument Mar 8 17:51:20 taft-01 lvm[3262]: Failed to suspend black_bird/synced_random_raid5_3legs_1 before committing changes Mar 8 17:51:20 taft-01 lvm[3262]: Failed to replace faulty devices in black_bird/synced_random_raid5_3legs_1. Mar 8 17:51:20 taft-01 lvm[3262]: Repair of RAID device black_bird-synced_random_raid5_3legs_1 failed. Mar 8 17:51:20 taft-01 lvm[3262]: Failed to process event for black_bird-synced_random_raid5_3legs_1 Version-Release number of selected component (if applicable): 2.6.32-220.4.2.el6.x86_64 lvm2-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 lvm2-libs-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 lvm2-cluster-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 udev-147-2.40.el6 BUILT: Fri Sep 23 07:51:13 CDT 2011 device-mapper-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-libs-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-event-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-event-libs-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 cmirror-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 How reproducible: Often
I have not been able to reproduce this since updating the kernel. Closing and will reopen if seen again in the future... 2.6.32-251.el6.x86_64 lvm2-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 lvm2-libs-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 lvm2-cluster-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 udev-147-2.40.el6 BUILT: Fri Sep 23 07:51:13 CDT 2011 device-mapper-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-libs-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-event-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 device-mapper-event-libs-1.02.74-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012 cmirror-2.02.95-1.el6 BUILT: Tue Mar 6 10:00:33 CST 2012