Bug 801571

Summary: manual raid volume repair can fail after single leg device failure
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.3CC: agk, dwysocha, heinzm, jbrassow, mbroz, mcsontos, msnitzer, prajnoha, prockai, syeghiay, thornber, zkabelac
Target Milestone: rcKeywords: Reopened, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.98-6.el6 Doc Type: Bug Fix
Doc Text:
Previously, if a device failed while a RAID logical volume was not in-sync, any attempts to fix it would fail in an uncontrolled way. This case is now handled, but there are some changes that must be noted: 1) You cannot repair or replace devices in a RAID logical volume that is not active. The tool ('lvconvert --repair') must know the sync status of the array and can only get that when the array is active. 2) You cannot replace a device in a RAID logical volume that has not completed its initial synchronization. Doing so would produce unpredictable results and is therefore disallowed. 3) You can repair a RAID logical volume that has not completed its initial synchronization, but some data may not be recoverable because it had not had time to make that data fully redundant. In this case, a warning is printed and the user is queried if they would like to proceed.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 08:03:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 895654    

Description Corey Marthaler 2012-03-08 21:34:47 UTC 
Description of problem:
Scenario kill_random_synced_raid6_3legs: Kill random leg of synced 3 leg raid6 volume(s)

********* RAID hash info for this scenario *********
* names:              synced_random_raid6_3legs_1
* sync:               1
* type:               raid6
* -m |-i value:       3
* leg devices:        /dev/sdd1 /dev/sdb1 /dev/sdg1 /dev/sdc1 /dev/sde1
* failpv(s):          /dev/sde1
* failnode(s):        taft-01
* raid fault policy:   warn
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid6 -i 3 -n synced_random_raid6_3legs_1 -L 500M black_bird /dev/sdd1:0-1000 /dev/sdb1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-1000 /dev/sde1:0-1000

RAID Structure(s):
  LV                                     Attr     LSize   Devices
  synced_random_raid6_3legs_1            rwi-a-r- 504.00m synced_random_raid6_3legs_1_rimage_0(0),synced_random_raid6_3legs_1_rimage_1(0),synced_random_raid6_3legs_1_rimage_2(0),synced_random_raid6_3legs_1_rimage_3(0),synced_random_raid6_3legs_1_rimage_4(0)
  [synced_random_raid6_3legs_1_rimage_0] Iwi-aor- 168.00m /dev/sdd1(1)
  [synced_random_raid6_3legs_1_rimage_1] Iwi-aor- 168.00m /dev/sdb1(1)
  [synced_random_raid6_3legs_1_rimage_2] Iwi-aor- 168.00m /dev/sdg1(1)
  [synced_random_raid6_3legs_1_rimage_3] Iwi-aor- 168.00m /dev/sdc1(1)
  [synced_random_raid6_3legs_1_rimage_4] Iwi-aor- 168.00m /dev/sde1(1)
  [synced_random_raid6_3legs_1_rmeta_0]  ewi-aor-   4.00m /dev/sdd1(0)
  [synced_random_raid6_3legs_1_rmeta_1]  ewi-aor-   4.00m /dev/sdb1(0)
  [synced_random_raid6_3legs_1_rmeta_2]  ewi-aor-   4.00m /dev/sdg1(0)
  [synced_random_raid6_3legs_1_rmeta_3]  ewi-aor-   4.00m /dev/sdc1(0)
  [synced_random_raid6_3legs_1_rmeta_4]  ewi-aor-   4.00m /dev/sde1(0)

PV=/dev/sde1
        synced_random_raid6_3legs_1_rimage_4: 2
        synced_random_raid6_3legs_1_rmeta_4: 2

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

Sleeping 10 seconds to get some outsanding EXT I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Disabling device sde on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.512046 s, 81.9 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sde1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sde1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sde1: read failed after 0 of 512 at 0: Input/output error
  /dev/sde1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sde1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid vPvSJq-6lYL-oLrB-ejdo-VsTJ-qR2H-zZScvg.
  LV                                     Attr     LSize   Copy%  Devices
  synced_random_raid6_3legs_1            rwi-aor- 504.00m        synced_random_raid6_3legs_1_rimage_0(0),synced_random_raid6_3legs_1_rimage_1(0),synced_random_raid6_3legs_1_rimage_2(0),synced_random_raid6_3legs_1_rimage_3(0),synced_random_raid6_3legs_1_rimage_4(0)
  [synced_random_raid6_3legs_1_rimage_0] iwi-aor- 168.00m        /dev/sdd1(1)
  [synced_random_raid6_3legs_1_rimage_1] iwi-aor- 168.00m        /dev/sdb1(1)
  [synced_random_raid6_3legs_1_rimage_2] iwi-aor- 168.00m        /dev/sdg1(1)
  [synced_random_raid6_3legs_1_rimage_3] iwi-aor- 168.00m        /dev/sdc1(1)
  [synced_random_raid6_3legs_1_rimage_4] iwi-aor- 168.00m        unknown device(1)
  [synced_random_raid6_3legs_1_rmeta_0]  ewi-aor-   4.00m        /dev/sdd1(0)
  [synced_random_raid6_3legs_1_rmeta_1]  ewi-aor-   4.00m        /dev/sdb1(0)
  [synced_random_raid6_3legs_1_rmeta_2]  ewi-aor-   4.00m        /dev/sdg1(0)
  [synced_random_raid6_3legs_1_rmeta_3]  ewi-aor-   4.00m        /dev/sdc1(0)
  [synced_random_raid6_3legs_1_rmeta_4]  ewi-aor-   4.00m        unknown device(0)

Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdc1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of synced_random_raid6_3legs_1_rimage_4 on:  taft-01
Checking EXISTENCE and STATE of synced_random_raid6_3legs_1_rmeta_4 on:  taft-01

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdd1 /dev/sdb1 /dev/sdg1 /dev/sdc1 unknown
ACTUAL LEG ORDER: /dev/sdd1 /dev/sdb1 /dev/sdg1 /dev/sdc1 unknown

Fault policy is warn, manually repairing failed raid volumes
taft-01: 'echo y | lvconvert --repair black_bird/synced_random_raid6_3legs_1'
Attempt to replace failed RAID images (requires full device resync)? [y/n]:   device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/synced_random_raid6_3legs_1 before committing changes
  Failed to replace faulty devices in black_bird/synced_random_raid6_3legs_1.
  Device '/dev/sdf1' has been left open.
  [...]
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/synced_random_raid6_3legs_1 on taft-01



taft-01 kernel: device-mapper: raid: Superblock area of rebuild device 4 should have been cleared.
taft-01 kernel: md/raid:mdX: device dm-10 operational as raid disk 3
taft-01 kernel: md/raid:mdX: device dm-8 operational as raid disk 2
taft-01 kernel: md/raid:mdX: device dm-6 operational as raid disk 1
taft-01 kernel: md/raid:mdX: device dm-4 operational as raid disk 0
taft-01 kernel: md/raid:mdX: allocated 5334kB
taft-01 kernel: md/raid:mdX: raid level 6 active with 4 out of 5 devices, algorithm 8
taft-01 kernel: mdX: bitmap file is out of date (37 < 56) -- forcing full recovery
taft-01 kernel: created bitmap (1 pages) for device mdX
taft-01 lvm[7935]: No longer monitoring RAID device black_bird-synced_random_raid6_3legs_1 for events.
taft-01 kernel: mdX: bitmap file is out of date, doing full recovery
taft-01 kernel: mdX: bitmap initialized from disk: read 1/1 pages, set 336 of 336 bits
taft-01 lvm[7935]: Monitoring RAID device black_bird-synced_random_raid6_3legs_1 for events.
taft-01 lvm[7935]: raid6_zr array, black_bird-synced_random_raid6_3legs_1, is not in-sync.
taft-01 kernel: md: recovery of RAID array mdX
taft-01 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
taft-01 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
taft-01 kernel: md: using 128k window, over a total of 172032k.
taft-01 kernel: device-mapper: table: 253:13: raid: Cannot change device positions in RAID array
taft-01 kernel: device-mapper: ioctl: error adding target to table
taft-01 kernel: md: mdX: recovery done.
taft-01 lvm[7935]: raid6_zr array, black_bird-synced_random_raid6_3legs_1, is now in-sync.


[root@taft-01 tmp]# lvs -a -o +devices
  Couldn't find device with uuid vPvSJq-6lYL-oLrB-ejdo-VsTJ-qR2H-zZScvg.
  LV                                             VG         Attr     LSize   Devices
  synced_random_raid6_3legs_1                    black_bird rwi-aor- 504.00m synced_random_raid6_3legs_1_rimage_0(0),synced_random_raid6_3legs_1_rimage_1(0),synced_random_raid6_3legs_1_rimage_2(0),synced_random_raid6_3legs_1_rimage_3(0),synced_random_raid6_3legs_1_rimage_5(0)
  [synced_random_raid6_3legs_1_rimage_0]         black_bird iwi-aor- 168.00m /dev/sdd1(1)
  [synced_random_raid6_3legs_1_rimage_1]         black_bird iwi-aor- 168.00m /dev/sdb1(1)
  [synced_random_raid6_3legs_1_rimage_2]         black_bird iwi-aor- 168.00m /dev/sdg1(1)
  [synced_random_raid6_3legs_1_rimage_3]         black_bird iwi-aor- 168.00m /dev/sdc1(1)
  synced_random_raid6_3legs_1_rimage_4_extracted black_bird -wi----- 168.00m unknown device(1)
  [synced_random_raid6_3legs_1_rimage_5]         black_bird iwi-aor- 168.00m /dev/sdh1(1)
  [synced_random_raid6_3legs_1_rmeta_0]          black_bird ewi-aor-   4.00m /dev/sdd1(0)
  [synced_random_raid6_3legs_1_rmeta_1]          black_bird ewi-aor-   4.00m /dev/sdb1(0)
  [synced_random_raid6_3legs_1_rmeta_2]          black_bird ewi-aor-   4.00m /dev/sdg1(0)
  [synced_random_raid6_3legs_1_rmeta_3]          black_bird ewi-aor-   4.00m /dev/sdc1(0)
  synced_random_raid6_3legs_1_rmeta_4_extracted  black_bird -wi-----   4.00m unknown device(0)
  [synced_random_raid6_3legs_1_rmeta_5]          black_bird ewi-aor-   4.00m /dev/sdh1(0)


Version-Release number of selected component (if applicable):
2.6.32-220.4.2.el6.x86_64

lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012


How reproducible:
30% of the time
Comment 1 Corey Marthaler 2012-03-08 21:52:06 UTC 
This happens with raid4 devices as well.

Fault policy is warn, manually repairing failed raid volumes
taft-01: 'echo y | lvconvert --repair black_bird/synced_primary_raid4_3legs_1'
  /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid NYUlem-sF4t-j1PC-Cd19-pLEx-B8LG-hlLGit.
Attempt to replace failed RAID images (requires full device resync)? [y/n]:   device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/synced_primary_raid4_3legs_1 before committing changes
  Failed to replace faulty devices in black_bird/synced_primary_raid4_3legs_1.
  Device '/dev/sdf1' has been left open.
  [...] 
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/synced_primary_raid4_3legs_1 on taft-01


Mar  8 15:44:35 taft-01 qarshd[3744]: Running cmdline: echo y | lvconvert --repair black_bird/synced_primary_raid4_3legs_1
Mar  8 15:44:36 taft-01 kernel: device-mapper: raid: Reshaping arrays not yet supported.
Mar  8 15:44:36 taft-01 kernel: device-mapper: table: 253:11: raid: Unable to assemble array: Invalid superblocks
Mar  8 15:44:36 taft-01 kernel: device-mapper: ioctl: error adding target to table


[root@taft-01 ~]# lvs -a -o +devices
  /dev/sdg1: read failed after 0 of 512 at 145669554176: Input/output error
  Couldn't find device with uuid NYUlem-sF4t-j1PC-Cd19-pLEx-B8LG-hlLGit.
  LV                                      VG         Attr     LSize   Devices
  synced_primary_raid4_3legs_1            black_bird rwi-aor- 504.00m synced_primary_raid4_3legs_1_rimage_0(0),synced_primary_raid4_3legs_1_rimage_1(0),synced_primary_raid4_3legs_1_rimage_2(0),synced_primary_raid4_3legs_1_rimage_3(0)
  [synced_primary_raid4_3legs_1_rimage_0] black_bird iwi-aor- 168.00m unknown device(1)
  [synced_primary_raid4_3legs_1_rimage_1] black_bird iwi-aor- 168.00m /dev/sdc1(1)
  [synced_primary_raid4_3legs_1_rimage_2] black_bird iwi-aor- 168.00m /dev/sdb1(1)
  [synced_primary_raid4_3legs_1_rimage_3] black_bird iwi-aor- 168.00m /dev/sde1(1)
  [synced_primary_raid4_3legs_1_rmeta_0]  black_bird ewi-aor-   4.00m unknown device(0)
  [synced_primary_raid4_3legs_1_rmeta_1]  black_bird ewi-aor-   4.00m /dev/sdc1(0)
  [synced_primary_raid4_3legs_1_rmeta_2]  black_bird ewi-aor-   4.00m /dev/sdb1(0)
  [synced_primary_raid4_3legs_1_rmeta_3]  black_bird ewi-aor-   4.00m /dev/sde1(0)
Comment 2 Corey Marthaler 2012-03-09 00:24:52 UTC 
The issue in comment #1 may actually be bug 801613. Actually, they may both be related.
Comment 3 Jonathan Earl Brassow 2012-03-12 19:56:40 UTC 
This works for me.  You mention in comment #1 that it is reproducible 30% of the time.  What is it that you are doing differently that 30% of the time that causes the failure?


[root@bp-01 ~]# devices vg
  LV            Copy%  Devices                                                                   
  lv                   lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0),lv_rimage_3(0),lv_rimage_4(0)
  [lv_rimage_0]        /dev/sdb1(1)                                                              
  [lv_rimage_1]        /dev/sdc1(1)                                                              
  [lv_rimage_2]        /dev/sdd1(1)                                                              
  [lv_rimage_3]        /dev/sde1(1)                                                              
  [lv_rimage_4]        /dev/sdf1(1)                                                              
  [lv_rmeta_0]         /dev/sdb1(0)                                                              
  [lv_rmeta_1]         /dev/sdc1(0)                                                              
  [lv_rmeta_2]         /dev/sdd1(0)                                                              
  [lv_rmeta_3]         /dev/sde1(0)                                                              
  [lv_rmeta_4]         /dev/sdf1(0)                                                              
[root@bp-01 ~]# off.sh sdc
Turning off sdc
[root@bp-01 ~]# lvconvert --repair vg/lv
  /dev/sdc1: read failed after 0 of 2048 at 250994294784: Input/output error
  /dev/sdc1: read failed after 0 of 2048 at 250994376704: Input/output error
  /dev/sdc1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sdc1: read failed after 0 of 2048 at 4096: Input/output error
  Couldn't find device with uuid sPkCdG-oM1j-IcIj-hlIa-4elN-g7N3-LU0993.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  Faulty devices in vg/lv successfully replaced.
[root@bp-01 ~]# devices vg
  Couldn't find device with uuid sPkCdG-oM1j-IcIj-hlIa-4elN-g7N3-LU0993.
  LV            Copy%  Devices                                                                   
  lv                   lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0),lv_rimage_3(0),lv_rimage_4(0)
  [lv_rimage_0]        /dev/sdb1(1)                                                              
  [lv_rimage_1]        /dev/sdg1(1)                                                              
  [lv_rimage_2]        /dev/sdd1(1)                                                              
  [lv_rimage_3]        /dev/sde1(1)                                                              
  [lv_rimage_4]        /dev/sdf1(1)                                                              
  [lv_rmeta_0]         /dev/sdb1(0)                                                              
  [lv_rmeta_1]         /dev/sdg1(0)                                                              
  [lv_rmeta_2]         /dev/sdd1(0)                                                              
  [lv_rmeta_3]         /dev/sde1(0)                                                              
  [lv_rmeta_4]         /dev/sdf1(0)
Comment 4 Corey Marthaler 2012-03-12 21:41:24 UTC 
I'll try and narrow this down, but is does still fail for me with the rpms that are in the official errata.

taft-01: 'echo y | lvconvert --repair black_bird/synced_random_raid6_3legs_1'
  /dev/sdb1: read failed after 0 of 512 at 145669554176: Input/output error
  [...]
  /dev/sdb1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid u15Y60-CtBL-4jdl-05L8-lM0n-ZvOZ-a6QHKH.
Attempt to replace failed RAID images (requires full device resync)? [y/n]:   device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/synced_random_raid6_3legs_1 before committing changes
  Failed to replace faulty devices in black_bird/synced_random_raid6_3legs_1.
  Device '/dev/sdf1' has been left open.
  [...]
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/synced_random_raid6_3legs_1 on taft-01


 kernel: device-mapper: raid: Superblock area of rebuild device 3 should have been cleared.
 kernel: md/raid:mdX: device dm-12 operational as raid disk 4
 kernel: md/raid:mdX: device dm-7 operational as raid disk 2
 kernel: md/raid:mdX: device dm-5 operational as raid disk 1
 kernel: md/raid:mdX: device dm-3 operational as raid disk 0
 kernel: md/raid:mdX: allocated 5334kB
 kernel: md/raid:mdX: raid level 6 active with 4 out of 5 devices, algorithm 8
 kernel: mdX: bitmap file is out of date (51 < 56) -- forcing full recovery
 kernel: created bitmap (1 pages) for device mdX
 lvm[1273]: No longer monitoring RAID device black_bird-synced_random_raid6_3legs_1 for events.
 kernel: mdX: bitmap file is out of date, doing full recovery
 kernel: mdX: bitmap initialized from disk: read 1/1 pages, set 336 of 336 bits
 lvm[1273]: Monitoring RAID device black_bird-synced_random_raid6_3legs_1 for events.
 lvm[1273]: raid6_zr array, black_bird-synced_random_raid6_3legs_1, is not in-sync.
 kernel: md: recovery of RAID array mdX
 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
 kernel: md: using 128k window, over a total of 172032k.
 kernel: sd 3:0:0:1: rejecting I/O to offline device
[...]
 kernel: sd 3:0:0:1: rejecting I/O to offline device
 kernel: device-mapper: table: 253:13: raid: Cannot change device positions in RAID array
 kernel: device-mapper: ioctl: error adding target to table
 kernel: md: mdX: recovery done.
 lvm[1273]: raid6_zr array, black_bird-synced_random_raid6_3legs_1, is now in-sync.


[root@taft-01 ~]# lvs -a -o +devices
/dev/sdb1: read failed after 0 of 512 at 145669554176: Input/output error
[...]
Couldn't find device with uuid u15Y60-CtBL-4jdl-05L8-lM0n-ZvOZ-a6QHKH.
LV                                             VG         Attr     LSize   Devices
synced_random_raid6_3legs_1                    black_bird rwi-aor- 504.00m synced_random_raid6_3legs_1_rimage_0(0),synced_random_raid6_3legs_1_rimage_1(0),synced_random_raid6_3legs_1_rimage_2(0),synced_random_raid6_3legs_1_rimage_5(0),synced_random_raid6_3legs_1_rimage_4(0)
[synced_random_raid6_3legs_1_rimage_0]         black_bird iwi-aor- 168.00m /dev/sde1(1)
[synced_random_raid6_3legs_1_rimage_1]         black_bird iwi-aor- 168.00m /dev/sdg1(1)
[synced_random_raid6_3legs_1_rimage_2]         black_bird iwi-aor- 168.00m /dev/sdf1(1)
synced_random_raid6_3legs_1_rimage_3_extracted black_bird -wi----- 168.00m unknown device(1)
[synced_random_raid6_3legs_1_rimage_4]         black_bird iwi-aor- 168.00m /dev/sdh1(1)
[synced_random_raid6_3legs_1_rimage_5]         black_bird iwi-aor- 168.00m /dev/sdd1(1)
[synced_random_raid6_3legs_1_rmeta_0]          black_bird ewi-aor-   4.00m /dev/sde1(0)
[synced_random_raid6_3legs_1_rmeta_1]          black_bird ewi-aor-   4.00m /dev/sdg1(0)
[synced_random_raid6_3legs_1_rmeta_2]          black_bird ewi-aor-   4.00m /dev/sdf1(0)
synced_random_raid6_3legs_1_rmeta_3_extracted  black_bird -wi-----   4.00m unknown device(0)
[synced_random_raid6_3legs_1_rmeta_4]          black_bird ewi-aor-   4.00m /dev/sdh1(0)
[synced_random_raid6_3legs_1_rmeta_5]          black_bird ewi-aor-   4.00m /dev/sdd1(0)

synced_random_raid6_3legs_2                    black_bird rwi-aor- 504.00m synced_random_raid6_3legs_2_rimage_0(0),synced_random_raid6_3legs_2_rimage_1(0),synced_random_raid6_3legs_2_rimage_2(0),synced_random_raid6_3legs_2_rimage_3(0),synced_random_raid6_3legs_2_rimage_4(0)
[synced_random_raid6_3legs_2_rimage_0]         black_bird iwi-aor- 168.00m /dev/sde1(44)
[synced_random_raid6_3legs_2_rimage_1]         black_bird iwi-aor- 168.00m /dev/sdg1(44)
[synced_random_raid6_3legs_2_rimage_2]         black_bird iwi-aor- 168.00m /dev/sdf1(44)
[synced_random_raid6_3legs_2_rimage_3]         black_bird iwi-aor- 168.00m unknown device(44)
[synced_random_raid6_3legs_2_rimage_4]         black_bird iwi-aor- 168.00m /dev/sdh1(44)
[synced_random_raid6_3legs_2_rmeta_0]          black_bird ewi-aor-   4.00m /dev/sde1(43)
[synced_random_raid6_3legs_2_rmeta_1]          black_bird ewi-aor-   4.00m /dev/sdg1(43)
[synced_random_raid6_3legs_2_rmeta_2]          black_bird ewi-aor-   4.00m /dev/sdf1(43)
[synced_random_raid6_3legs_2_rmeta_3]          black_bird ewi-aor-   4.00m unknown device(43)
[synced_random_raid6_3legs_2_rmeta_4]          black_bird ewi-aor-   4.00m /dev/sdh1(43)


2.6.32-220.4.2.el6.x86_64
lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
Comment 5 Corey Marthaler 2012-03-14 22:36:14 UTC 
I have not been able to reproduce this since updating the kernel. Closing and will reopen if seen again in the future...

2.6.32-251.el6.x86_64
lvm2-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-libs-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
lvm2-cluster-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
device-mapper-event-libs-1.02.74-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
cmirror-2.02.95-1.el6    BUILT: Tue Mar  6 10:00:33 CST 2012
Comment 6 Corey Marthaler 2012-11-07 17:10:09 UTC 
I'm seeing this again while testing RAID10 with the latest 6.4 rpms.

./black_bird -l /home/msp/cmarthal/work/sts/sts-root -o taft-01 -r /usr/tests/sts-rhel6.4 -e kill_random_non_synced_raid10_3legs 

Scenario kill_random_non_synced_raid10_3legs: Kill random leg of NON synced 3 leg raid10 volume(s)

********* RAID hash info for this scenario *********
* names:              non_synced_random_raid10_3legs_1
* sync:               0
* type:               raid10
* -m |-i value:       3
* leg devices:        /dev/sdc1 /dev/sde1 /dev/sdb1 /dev/sdf1 /dev/sdd1 /dev/sdh1
* failpv(s):          /dev/sdf1
* failnode(s):        taft-01
* raid fault policy:   warn
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid10 -i 3 -n non_synced_random_raid10_3legs_1 -L 1.5G black_bird /dev/sdc1:0-1000 /dev/sde1:0-1000 /dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdd1:0-1000 /dev/sdh1:0-1000

RAID Structure(s):
  LV                                          Attr      LSize   Copy%  Devices
  non_synced_random_raid10_3legs_1            rwi-a-r--   1.50g   6.25 non_synced_random_raid10_3legs_1_rimage_0(0),non_synced_random_raid10_3legs_1_rimage_1(0),non_synced_random_raid10_3legs_1_rimage_2(0),non_synced_random_raid10_3legs_1_rimage_3(0),non_synced_random_raid10_3legs_1_rimage_4(0),non_synced_random_raid10_3legs_1_rimage_5(0)
  [non_synced_random_raid10_3legs_1_rimage_0] Iwi-aor-- 512.00m        /dev/sdc1(1)
  [non_synced_random_raid10_3legs_1_rimage_1] Iwi-aor-- 512.00m        /dev/sde1(1)
  [non_synced_random_raid10_3legs_1_rimage_2] Iwi-aor-- 512.00m        /dev/sdb1(1)
  [non_synced_random_raid10_3legs_1_rimage_3] Iwi-aor-- 512.00m        /dev/sdf1(1)
  [non_synced_random_raid10_3legs_1_rimage_4] Iwi-aor-- 512.00m        /dev/sdd1(1)
  [non_synced_random_raid10_3legs_1_rimage_5] Iwi-aor-- 512.00m        /dev/sdh1(1)
  [non_synced_random_raid10_3legs_1_rmeta_0]  ewi-aor--   4.00m        /dev/sdc1(0)
  [non_synced_random_raid10_3legs_1_rmeta_1]  ewi-aor--   4.00m        /dev/sde1(0)
  [non_synced_random_raid10_3legs_1_rmeta_2]  ewi-aor--   4.00m        /dev/sdb1(0)
  [non_synced_random_raid10_3legs_1_rmeta_3]  ewi-aor--   4.00m        /dev/sdf1(0)
  [non_synced_random_raid10_3legs_1_rmeta_4]  ewi-aor--   4.00m        /dev/sdd1(0)
  [non_synced_random_raid10_3legs_1_rmeta_5]  ewi-aor--   4.00m        /dev/sdh1(0)

PV=/dev/sdf1
        non_synced_random_raid10_3legs_1_rimage_3: 2
        non_synced_random_raid10_3legs_1_rmeta_3: 2

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

<start name="taft-01_non_synced_random_raid10_3legs_1" pid="1245" time="Wed Nov  7 10:43:43 2012" type="cmd" />
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Current sync percent just before failure
        ( 26.86% )
Disabling device sdf on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.241206 s, 174 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  [...]
  /dev/sdf1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid EpElLL-p3oh-17c9-Irz7-Xdu8-f1yG-n5wmPl.
  LV                                          Attr      LSize   Copy%  Devices
  non_synced_random_raid10_3legs_1            rwi-aor-p   1.50g  71.88 non_synced_random_raid10_3legs_1_rimage_0(0),non_synced_random_raid10_3legs_1_rimage_1(0),non_synced_random_raid10_3legs_1_rimage_2(0),non_synced_random_raid10_3legs_1_rimage_3(0),non_synced_random_raid10_3legs_1_rimage_4(0),non_synced_random_raid10_3legs_1_rimage_5(0)
  [non_synced_random_raid10_3legs_1_rimage_0] Iwi-aor-- 512.00m        /dev/sdc1(1)
  [non_synced_random_raid10_3legs_1_rimage_1] Iwi-aor-- 512.00m        /dev/sde1(1)
  [non_synced_random_raid10_3legs_1_rimage_2] Iwi-aor-- 512.00m        /dev/sdb1(1)
  [non_synced_random_raid10_3legs_1_rimage_3] Iwi-aor-p 512.00m        unknown device(1)
  [non_synced_random_raid10_3legs_1_rimage_4] Iwi-aor-- 512.00m        /dev/sdd1(1)
  [non_synced_random_raid10_3legs_1_rimage_5] Iwi-aor-- 512.00m        /dev/sdh1(1)
  [non_synced_random_raid10_3legs_1_rmeta_0]  ewi-aor--   4.00m        /dev/sdc1(0)
  [non_synced_random_raid10_3legs_1_rmeta_1]  ewi-aor--   4.00m        /dev/sde1(0)
  [non_synced_random_raid10_3legs_1_rmeta_2]  ewi-aor--   4.00m        /dev/sdb1(0)
  [non_synced_random_raid10_3legs_1_rmeta_3]  ewi-aor-p   4.00m        unknown device(0)
  [non_synced_random_raid10_3legs_1_rmeta_4]  ewi-aor--   4.00m        /dev/sdd1(0)
  [non_synced_random_raid10_3legs_1_rmeta_5]  ewi-aor--   4.00m        /dev/sdh1(0)

Verifying FAILED device /dev/sdf1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdc1 *IS* in the volume(s)
Verifying IMAGE device /dev/sde1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdh1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of non_synced_random_raid10_3legs_1_rimage_3 on:  taft-01
Checking EXISTENCE and STATE of non_synced_random_raid10_3legs_1_rmeta_3 on:  taft-01

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdc1 /dev/sde1 /dev/sdb1 unknown /dev/sdd1 /dev/sdh1
ACTUAL LEG ORDER: /dev/sdc1 /dev/sde1 /dev/sdb1 unknown /dev/sdd1 /dev/sdh1
/dev/sdc1 ne /dev/sdc1
/dev/sde1 ne /dev/sde1
/dev/sdb1 ne /dev/sdb1
unknown ne unknown
/dev/sdd1 ne /dev/sdd1
/dev/sdh1 ne /dev/sdh1

Fault policy is warn, manually repairing failed raid volumes
taft-01: 'lvconvert --yes --repair black_bird/non_synced_random_raid10_3legs_1'
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdf1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid EpElLL-p3oh-17c9-Irz7-Xdu8-f1yG-n5wmPl.
  device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/non_synced_random_raid10_3legs_1 before committing changes
  Failed to replace faulty devices in black_bird/non_synced_random_raid10_3legs_1.
  Device '/dev/sdb1' has been left open.
  [...]
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/non_synced_random_raid10_3legs_1 on taft-01
Comment 7 Jonathan Earl Brassow 2012-11-28 17:20:04 UTC 
Seems to work just fine for me.  This is with kernel-2.6.32-344.el6.x86_64 and the latest lvm2/device-mapper sources.  I don't see RAID bug fixes (like 866678) in the latest lvm2 RPMs though.  You'll have to look for an LVM package > 11/24/2012.

================================================================================
Iteration 16.1 started at Wed Nov 28 11:10:32 CST 2012
================================================================================
Scenario kill_random_non_synced_raid10_3legs: Kill random leg of NON synced 3 leg raid10 volume(s)

********* RAID hash info for this scenario *********
* names:              non_synced_random_raid10_3legs_1 non_synced_random_raid10_3legs_2
* sync:               0
* type:               raid10
* -m |-i value:       3
* leg devices:        /dev/sdc1 /dev/sde1 /dev/sdf1 /dev/sdd1 /dev/sdh1 /dev/sdb1
* failpv(s):          /dev/sde1
* failnode(s):        bp-01
* additional snap:    /dev/sdc1
* raid fault policy:   warn
******************************************************

Please verify fix with latest RPMs > 11/24/2012.
Comment 9 Marian Csontos 2012-12-05 09:03:53 UTC 
Corey, any chance dmeventd is coredumping there - Bug 879646 ?
Comment 10 Corey Marthaler 2012-12-06 19:49:17 UTC 
Marian, 

Nope, I just hit this again today, and dmeventd is still running after the failure.

2.6.32-343.el6.x86_64
lvm2-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
lvm2-libs-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
lvm2-cluster-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
device-mapper-libs-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
device-mapper-event-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
device-mapper-event-libs-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
cmirror-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012


================================================================================
Iteration 0.1 started at Thu Dec  6 13:12:02 CST 2012
================================================================================
Scenario kill_random_non_synced_raid1_2legs: Kill random leg of NON synced 2 leg raid1 volume(s)

********* RAID hash info for this scenario *********
* names:              non_synced_random_raid1_2legs_1
* sync:               0
* type:               raid1
* -m |-i value:       2
* leg devices:        /dev/sdg1 /dev/sdf1 /dev/sdc1
* failpv(s):          /dev/sdf1
* failnode(s):        taft-01
* raid fault policy:   warn
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid1 -m 2 -n non_synced_random_raid1_2legs_1 -L 1.5G black_bird /dev/sdg1:0-1000 /dev/sdf1:0-1000 /dev/sdc1:0-1000

RAID Structure(s):
  LV                                         Attr      LSize   Cpy%Sync Devices
  non_synced_random_raid1_2legs_1            rwi-a-r--   1.50g     0.00 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
  [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g          /dev/sdg1(1)
  [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor--   1.50g          /dev/sdf1(1)
  [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor--   1.50g          /dev/sdc1(1)
  [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m          /dev/sdg1(0)
  [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m          /dev/sdf1(0)
  [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor--   4.00m          /dev/sdc1(0)

PV=/dev/sdf1
        non_synced_random_raid1_2legs_1_rimage_1: 2
        non_synced_random_raid1_2legs_1_rmeta_1: 2

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Current sync percent just before failure
        ( 18.20% )
Disabling device sdf on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.237119 s, 177 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  [...]
  Couldn't find device with uuid cVo5f0-eq2W-MCog-Sfj1-2eBG-9bnj-poomOM.
  LV                                         Attr      LSize   Cpy%Sync Devices
  non_synced_random_raid1_2legs_1            rwi-aor-p   1.50g    38.02 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
  [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g          /dev/sdg1(1)
  [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor-p   1.50g          unknown device(1)
  [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor--   1.50g          /dev/sdc1(1)
  [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m          /dev/sdg1(0)
  [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor-p   4.00m          unknown device(0)
  [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor--   4.00m          /dev/sdc1(0)

Verifying FAILED device /dev/sdf1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdc1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of non_synced_random_raid1_2legs_1_rimage_1 on:  taft-01
Checking EXISTENCE and STATE of non_synced_random_raid1_2legs_1_rmeta_1 on:  taft-01

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdg1 unknown /dev/sdc1
ACTUAL LEG ORDER: /dev/sdg1 unknown /dev/sdc1
/dev/sdg1 ne /dev/sdg1
unknown ne unknown
/dev/sdc1 ne /dev/sdc1
Fault policy is warn, manually repairing failed raid volumes
taft-01: 'lvconvert --yes --repair black_bird/non_synced_random_raid1_2legs_1'
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  [...]
  Couldn't find device with uuid cVo5f0-eq2W-MCog-Sfj1-2eBG-9bnj-poomOM.
  device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/non_synced_random_raid1_2legs_1 before committing changes
  Failed to replace faulty devices in black_bird/non_synced_random_raid1_2legs_1.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdc1' has been left open.
  [...]
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/non_synced_random_raid1_2legs_1 on taft-01

[root@taft-01 ~]# ps -ef | grep dmeventd
root      3122     1  0 Dec05 ?        00:00:40 /sbin/dmeventd

Dec  6 13:13:12 taft-01 lvm[3122]: Use 'lvconvert --repair black_bird/non_synced_random_raid1_2legs_1' to replace failed device

[root@taft-01 ~]# lvconvert --repair black_bird/non_synced_random_raid1_2legs_1
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdf1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid cVo5f0-eq2W-MCog-Sfj1-2eBG-9bnj-poomOM.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  device-mapper: create ioctl on black_bird-non_synced_random_raid1_2legs_1_rmeta_3 failed: Device or resource busy
  Failed to suspend black_bird/non_synced_random_raid1_2legs_1 before committing changes
  Failed to replace faulty devices in black_bird/non_synced_random_raid1_2legs_1.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdg1' has been left open.
  [...]
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
[root@taft-01 ~]# echo $?
5
Comment 11 Peter Rajnoha 2012-12-11 11:01:35 UTC 
(In reply to comment #7)
> Seems to work just fine for me.  This is with kernel-2.6.32-344.el6.x86_64
> and the latest lvm2/device-mapper sources.  I don't see RAID bug fixes (like
> 866678) in the latest lvm2 RPMs though.  You'll have to look for an LVM
> package > 11/24/2012.
...
> Please verify fix with latest RPMs > 11/24/2012.


(In reply to comment #10)
> Nope, I just hit this again today, and dmeventd is still running after the
> failure.
> 
> 2.6.32-343.el6.x86_64
> lvm2-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> lvm2-libs-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> lvm2-cluster-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
> device-mapper-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> device-mapper-libs-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> device-mapper-event-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> device-mapper-event-libs-1.02.77-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012
> cmirror-2.02.98-3.el6    BUILT: Mon Nov  5 06:45:48 CST 2012

The latest build is lvm2-2.02.98-4.el6 from 5th Dec, this one includes a few of the mirror patches. Corey, can you try this one as Jon suggested and if it still fails, we need to put this back to ASSIGNED then.
Comment 12 Marian Csontos 2012-12-11 11:55:07 UTC 
Corey, I believe, this is case where `lvconvert --repair` was called while raid was not synchronized yet. I noticed the RAID test cases for allocate policy failing due to not waiting for sync and suspect this is the same case - wait_for_sync is called only after `lvconvert --repair`.

Jon, was the operation disabled recently? (BZ?)
Comment 13 Corey Marthaler 2012-12-14 23:19:08 UTC 
No difference for me with the latest rpms. Still fails in the exact same way.

2.6.32-343.el6.x86_64
lvm2-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-libs-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-cluster-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
cmirror-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012

================================================================================
Iteration 0.1 started at Fri Dec 14 17:02:45 CST 2012
================================================================================
Scenario kill_random_non_synced_raid1_2legs: Kill random leg of NON synced 2 leg raid1 volume(s)

********* RAID hash info for this scenario *********
* names:              non_synced_random_raid1_2legs_1
* sync:               0
* type:               raid1
* -m |-i value:       2
* leg devices:        /dev/sdb1 /dev/sdd1 /dev/sdf1
* failpv(s):          /dev/sdf1
* failnode(s):        taft-01
* additional snap:    /dev/sdb1
* raid fault policy:   warn
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid1 -m 2 -n non_synced_random_raid1_2legs_1 -L 1.5G black_bird /dev/sdb1:0-1000 /dev/sdd1:0-1000 /dev/sdf1:0-1000

RAID Structure(s):
 LV                                         Attr      LSize   Cpy%Sync Devices
 non_synced_random_raid1_2legs_1            rwi-a-r--   1.50g     0.00 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
 [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g          /dev/sdb1(1)
 [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor--   1.50g          /dev/sdd1(1)
 [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor--   1.50g          /dev/sdf1(1)
 [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m          /dev/sdb1(0)
 [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m          /dev/sdd1(0)
 [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor--   4.00m          /dev/sdf1(0)

PV=/dev/sdf1
        non_synced_random_raid1_2legs_1_rimage_2: 2
        non_synced_random_raid1_2legs_1_rmeta_2: 2

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Creating a snapshot volume of each of the raids
Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----

Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Current sync percent just before failure
        ( 16.87% )
Disabling device sdf on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.240168 s, 175 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdf1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 9aVC5k-jZ2R-U6y6-Lq59-WcRr-FXG3-vDDY22.
  LV                                         Attr      LSize   Cpy%Sync Devices
  bb_snap1                                   swi-a-s-p 252.00m          /dev/sdb1(385)
  non_synced_random_raid1_2legs_1            owi-aor-p   1.50g    39.84 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
  [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g          /dev/sdb1(1)
  [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor--   1.50g          /dev/sdd1(1)
  [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor-p   1.50g          unknown device(1)
  [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m          /dev/sdb1(0)
  [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m          /dev/sdd1(0)
  [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor-p   4.00m          unknown device(0)

Verifying FAILED device /dev/sdf1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdd1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of non_synced_random_raid1_2legs_1_rimage_2 on:  taft-01
Checking EXISTENCE and STATE of non_synced_random_raid1_2legs_1_rmeta_2 on:  taft-01

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: /dev/sdb1 /dev/sdd1 unknown
ACTUAL LEG ORDER: /dev/sdb1 /dev/sdd1 unknown
/dev/sdb1 ne /dev/sdb1
/dev/sdd1 ne /dev/sdd1
unknown ne unknown
Fault policy is warn, manually repairing failed raid volumes
taft-01: 'lvconvert --yes --repair black_bird/non_synced_random_raid1_2legs_1'
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  [...]
  Couldn't find device with uuid 9aVC5k-jZ2R-U6y6-Lq59-WcRr-FXG3-vDDY22.
  device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend black_bird/non_synced_random_raid1_2legs_1 before committing changes
  Failed to replace faulty devices in black_bird/non_synced_random_raid1_2legs_1.
  Device '/dev/sdb1' has been left open.
  [...]
  Device '/dev/sdd1' has been left open.
lvconvert repair failed for black_bird/non_synced_random_raid1_2legs_1 on taft-01

[root@taft-01 ~]# lvconvert --repair black_bird/non_synced_random_raid1_2legs_1
  /dev/sdf1: read failed after 0 of 512 at 145669554176: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 145669664768: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdf1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdf1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid 9aVC5k-jZ2R-U6y6-Lq59-WcRr-FXG3-vDDY22.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  device-mapper: create ioctl on black_bird-non_synced_random_raid1_2legs_1_rmeta_3 failed: Device or resource busy
  Failed to suspend black_bird/non_synced_random_raid1_2legs_1 before committing changes
  Failed to replace faulty devices in black_bird/non_synced_random_raid1_2legs_1.
  Device '/dev/sdb1' has been left open.
  [...]
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
[root@taft-01 ~]# echo $?
5
Comment 14 Corey Marthaler 2012-12-14 23:26:36 UTC 
Now, if I run this same scenario with a raid_fault_policy set to 'allocate' instead of 'warn', dmeventd ends up segfaulting as mentioned in comment #9.

2.6.32-343.el6.x86_64
lvm2-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-libs-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-cluster-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
cmirror-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012

================================================================================
Iteration 0.1 started at Fri Dec 14 16:49:04 CST 2012
================================================================================
Scenario kill_random_non_synced_raid1_2legs: Kill random leg of NON synced 2 leg raid1 volume(s)

********* RAID hash info for this scenario *********
* names:              non_synced_random_raid1_2legs_1
* sync:               0
* type:               raid1
* -m |-i value:       2
* leg devices:        /dev/sdg1 /dev/sdb1 /dev/sde1
* failpv(s):          /dev/sde1
* failnode(s):        taft-01
* raid fault policy:   allocate
******************************************************

Creating raids(s) on taft-01...
taft-01: lvcreate --type raid1 -m 2 -n non_synced_random_raid1_2legs_1 -L 1.5G black_bird /dev/sdg1:0-1000 /dev/sdb1:0-1000 /dev/sde1:0-1000

RAID Structure(s):
 LV                                         Attr      LSize  Cpy%Sync Devices
 non_synced_random_raid1_2legs_1            rwi-a-r--   1.50g    0.00 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
 [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g         /dev/sdg1(1)
 [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor--   1.50g         /dev/sdb1(1)
 [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor--   1.50g         /dev/sde1(1)
 [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m         /dev/sdg1(0)
 [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m         /dev/sdb1(0)
 [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor--   4.00m         /dev/sde1(0)

PV=/dev/sde1
        non_synced_random_raid1_2legs_1_rimage_2: 1.0
        non_synced_random_raid1_2legs_1_rmeta_2: 1.0

Creating ext on top of mirror(s) on taft-01...
mke2fs 1.41.12 (17-May-2010)
Mounting mirrored ext filesystems on taft-01...

Writing verification files (checkit) to mirror(s) on...
        ---- taft-01 ----
Verifying files (checkit) on mirror(s) on...
        ---- taft-01 ----

Current sync percent just before failure
        ( 12.88% )
Disabling device sde on taft-01

Attempting I/O to cause mirror down conversion(s) on taft-01
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.233754 s, 179 MB/s

Verifying current sanity of lvm after the failure

RAID Structure(s):
 /dev/sde1: read failed after 0 of 512 at 145669554176: Input/output error
 [...]
 Couldn't find device with uuid OmsLQg-3ddW-Ombh-aZKw-YYHO-fQDJ-MbuWtA.
 LV                                         Attr      LSize  Cpy%Sync Devices
 non_synced_random_raid1_2legs_1            rwi-aor-p   1.50g   44.27 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
 [non_synced_random_raid1_2legs_1_rimage_0] Iwi-aor--   1.50g         /dev/sdg1(1)
 [non_synced_random_raid1_2legs_1_rimage_1] Iwi-aor--   1.50g         /dev/sdb1(1)
 [non_synced_random_raid1_2legs_1_rimage_2] Iwi-aor-p   1.50g         unknown device(1)
 [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m         /dev/sdg1(0)
 [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m         /dev/sdb1(0)
 [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor-p   4.00m         unknown device(0)

Verifying FAILED device /dev/sde1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdb1 *IS* in the volume(s)
verify the rimage/rmeta dm devices remain after the failures
Checking EXISTENCE and STATE of non_synced_random_raid1_2legs_1_rimage_2 on:  taft-01there should not be an 'unknown' device associated with non_synced_random_raid1_2legs_1_rimage_2 on taft-01
FI_engine: recover() method failed


Dec 14 16:49:27 taft-01 qarshd[3616]: Running cmdline: echo offline > /sys/block/sde/device/state &
Dec 14 16:49:27 taft-01 kernel: md: super_written gets error=-5, uptodate=0
Dec 14 16:49:27 taft-01 kernel: md/raid1:mdX: Disk failure on dm-7, disabling device.
Dec 14 16:49:27 taft-01 kernel: md/raid1:mdX: Operation continuing on 2 devices.
Dec 14 16:49:27 taft-01 lvm[1229]: Device #2 of raid1 array, black_bird-non_synced_random_raid1_2legs_1, has failed.
Dec 14 16:49:27 taft-01 kernel: md: mdX: resync done.
Dec 14 16:49:27 taft-01 kernel: sd 3:0:0:4: rejecting I/O to offline device
[...]
Dec 14 16:49:27 taft-01 kernel: sd 3:0:0:4: rejecting I/O to offline device
Dec 14 16:49:27 taft-01 kernel: md: checkpointing resync of mdX.
Dec 14 16:49:27 taft-01 kernel: md: resync of RAID array mdX
Dec 14 16:49:27 taft-01 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Dec 14 16:49:27 taft-01 kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Dec 14 16:49:27 taft-01 kernel: md: using 128k window, over a total of 1572864k.
Dec 14 16:49:27 taft-01 kernel: device-mapper: dm-log-userspace: version 1.1.0 loaded
[...]
Dec 14 16:49:27 taft-01 kernel: sd 3:0:0:4: rejecting I/O to offline device
Dec 14 16:49:28 taft-01 kernel: device-mapper: raid: Device 2 specified for rebuild: Clearing superblock
Dec 14 16:49:28 taft-01 kernel: device-mapper: raid: 'rebuild' specified while array is not in-sync
Dec 14 16:49:28 taft-01 kernel: device-mapper: table: 253:8: raid: Unable to assemble array: Invalid superblocks
Dec 14 16:49:28 taft-01 kernel: device-mapper: ioctl: error adding target to table
Dec 14 16:49:28 taft-01 lvm[1229]: device-mapper: reload ioctl on  failed: Invalid argument
Dec 14 16:49:28 taft-01 lvm[1229]: Failed to suspend black_bird/non_synced_random_raid1_2legs_1 before committing changes
Dec 14 16:49:28 taft-01 lvm[1229]: Failed to replace faulty devices in black_bird/non_synced_random_raid1_2legs_1.
Dec 14 16:49:28 taft-01 lvm[1229]: Device '/dev/sdf1' has been left open.
[...]
Dec 14 16:49:28 taft-01 lvm[1229]: Device '/dev/sdd1' has been left open.
Dec 14 16:49:28 taft-01 lvm[1229]: Repair of RAID device black_bird-non_synced_random_raid1_2legs_1 failed.
Dec 14 16:49:28 taft-01 lvm[1229]: Failed to process event for black_bird-non_synced_random_raid1_2legs_1
Dec 14 16:49:28 taft-01 lvm[1229]: Device #2 of raid1 array, black_bird-non_synced_random_raid1_2legs_1, has failed.
Dec 14 16:49:28 taft-01 kernel: dmeventd[3629]: segfault at 8 ip 0000003ebe2073d7 sp 00007fe9d92f5448 error 6 in libdevmapper.so.1.02[3ebe200000+36000]
Dec 14 16:49:28 taft-01 lvm[1229]: Child 3629 exited abnormally
Dec 14 16:49:28 taft-01 kernel: dmeventd[3547]: segfault at 8 ip 0000003ebe2073d7 sp 00007fe9d92f7738 error 6 in libdevmapper.so.1.02[3ebe200000+36000]
Dec 14 16:49:32 taft-01 xinetd[1833]: START: qarsh pid=3646 from=::ffff:10.15.80.47
Dec 14 16:49:32 taft-01 qarshd[3646]: Talking to peer 10.15.80.47:34740 (IPv4)
Dec 14 16:49:32 taft-01 qarshd[3646]: Running cmdline: pvs -a
Dec 14 16:49:32 taft-01 kernel: sd 3:0:0:4: rejecting I/O to offline device


[root@taft-01 ~]# lvs -a -o +devices
 /dev/sde1: read failed after 0 of 512 at 145669554176: Input/output error
 Couldn't find device with uuid OmsLQg-3ddW-Ombh-aZKw-YYHO-fQDJ-MbuWtA.
 LV                                         Attr      LSize   Cpy%Sync Devices
 non_synced_random_raid1_2legs_1            rwi-aor-p   1.50g   100.00 non_synced_random_raid1_2legs_1_rimage_0(0),non_synced_random_raid1_2legs_1_rimage_1(0),non_synced_random_raid1_2legs_1_rimage_2(0)
 [non_synced_random_raid1_2legs_1_rimage_0] iwi-aor--   1.50g          /dev/sdg1(1)
 [non_synced_random_raid1_2legs_1_rimage_1] iwi-aor--   1.50g          /dev/sdb1(1)
 [non_synced_random_raid1_2legs_1_rimage_2] iwi-aor-p   1.50g          unknown device(1)
 [non_synced_random_raid1_2legs_1_rmeta_0]  ewi-aor--   4.00m          /dev/sdg1(0)
 [non_synced_random_raid1_2legs_1_rmeta_1]  ewi-aor--   4.00m          /dev/sdb1(0)
 [non_synced_random_raid1_2legs_1_rmeta_2]  ewi-aor-p   4.00m          unknown device(0)
Comment 16 Jonathan Earl Brassow 2012-12-18 19:21:20 UTC 
This bug seems to be all about cases where devices are replaced before the array has had a chance to sync.  The kernel rejects this condition (unless forced) because if the array was not in-sync, it cannot gaurantee that the new device will be populated with correct data.  The kernel rejects the new DM table and the command dies early and uncleanly.  The 'not-in-sync' condition should be trapped.

Additionally, this problem does not just affect failure cases, but any condition that would try to replace a device while the array is not in-sync.  Here is another example:

[root@bp-01 ~]# lvcreate --type raid1 -m 2 -L 100G -n lv vg
  Logical volume "lv" created
[root@bp-01 ~]# devices vg
  LV            Cpy%Sync Devices                                     
  lv                0.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]          /dev/sda1(1)                                
  [lv_rimage_1]          /dev/sdb1(1)                                
  [lv_rimage_2]          /dev/sdc1(1)                                
  [lv_rmeta_0]           /dev/sda1(0)                                
  [lv_rmeta_1]           /dev/sdb1(0)                                
  [lv_rmeta_2]           /dev/sdc1(0)                                
[root@bp-01 ~]# lvconvert --replace /dev/sdc1 vg/lv
  device-mapper: reload ioctl on  failed: Invalid argument
  Failed to suspend vg/lv before committing changes
  Device '/dev/sdh1' has been left open.
  Device '/dev/sdh1' has been left open.
  ...
Comment 18 Jonathan Earl Brassow 2012-12-18 20:41:26 UTC 
Lessons learned:
 - Replacing devices should not be allowed if the array is not in-sync
 - Checking the sync status forces us to limit replacement to arrays that are active
 -- Again, NEVER allow replacement of devs in non-active array
 -- Accidentally replacing a device in an array which is inactive and not in-sync can cause LV to be inactivatable.
 - For repair, we can allow replacement of devices in an activate but not in-sync array; but a WARNING must be given - some data may be unrecoverable.

UNIT TESTS:
- changes pass LVM tests suite

DO NOT ALLOW REPLACEMENT WHEN ARRAY IS NOT ACTIVE:
[root@bp-01 lvm2]# devices vg
  LV            Cpy%Sync Devices                                     
  lv                7.59 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]          /dev/sda1(1)                                
  [lv_rimage_1]          /dev/sdb1(1)                                
  [lv_rimage_2]          /dev/sdd1(1)                                
  [lv_rmeta_0]           /dev/sda1(0)                                
  [lv_rmeta_1]           /dev/sdb1(0)                                
  [lv_rmeta_2]           /dev/sdd1(0)                                
[root@bp-01 lvm2]# vgchange -an vg
  0 logical volume(s) in volume group "vg" now active
[root@bp-01 lvm2]# lvconvert --replace /dev/sdd1 vg/lv
  vg/lv must be active to perform this operation.


DO NOT ALLOW REPLACEMENT OF DEVICES WHEN ARRAY IS NOT IN-SYNC:
[root@bp-01 lvm2]# lvcreate --type raid1 -m 2 -L 100G -n lv vg
  Logical volume "lv" created
[root@bp-01 lvm2]# devices vg
  LV            Cpy%Sync Devices                                     
  lv                0.34 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]          /dev/sda1(1)                                
  [lv_rimage_1]          /dev/sdb1(1)                                
  [lv_rimage_2]          /dev/sdc1(1)                                
  [lv_rmeta_0]           /dev/sda1(0)                                
  [lv_rmeta_1]           /dev/sdb1(0)                                
  [lv_rmeta_2]           /dev/sdc1(0)                                
[root@bp-01 lvm2]# lvconvert --replace /dev/sdc1 vg/lv
  Unable to replace devices in vg/lv while it is not in-sync.


ALLOW REPAIR, BUT WARN THAT PORTIONS OF ARRAY MAY BE UNRECOVERABLE:
[root@bp-01 lvm2]# off.sh sdc1
Turning off sdc
[root@bp-01 lvm2]# lvconvert --repair vg/lv
  /dev/sdc1: read failed after 0 of 512 at 898381381632: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 898381488128: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdc1: read failed after 0 of 512 at 4096: Input/output error
  /dev/sdc1: read failed after 0 of 2048 at 0: Input/output error
  Couldn't find device with uuid ZLrWFe-2ENN-S39Y-YI0b-oePB-WEc5-kLHnbL.
  WARNING: vg/lv is not in-sync.
  WARNING: Portions of the array may be unrecoverable.
Attempt to replace failed RAID images (requires full device resync)? [y/n]: y
  Faulty devices in vg/lv successfully replaced.

Non-manual (dmeventd) repair is the same.  If set to "allocate" replacement devices, then the devices will be replace but the warnings will be printed:
----------------------------- /var/log/messages ---------------------------
Dec 18 13:55:00 bp-01 lvm[10557]: WARNING: vg/lv is not in-sync.
Dec 18 13:55:00 bp-01 lvm[10557]: WARNING: Portions of the array may be unrecove
rable.
Dec 18 13:55:01 bp-01 kernel: device-mapper: raid: Device 2 specified for rebuil
d: Clearing superblock
Dec 18 13:55:01 bp-01 kernel: md/raid1:mdX: active with 2 out of 3 mirrors
Dec 18 13:55:01 bp-01 kernel: created bitmap (100 pages) for device mdX
Dec 18 13:55:01 bp-01 kernel: md: md_do_sync() got signal ... exiting
Dec 18 13:55:01 bp-01 kernel: md: checkpointing resync of mdX.
Dec 18 13:55:01 bp-01 kernel: mdX: bitmap initialized from disk: read 7 pages, s
et 201692 of 204800 bits
Dec 18 13:55:01 bp-01 lvm[10557]: Monitoring RAID device vg-lv for events.
Dec 18 13:55:01 bp-01 kernel: md: recovery of RAID array mdX
Dec 18 13:55:01 bp-01 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Dec 18 13:55:01 bp-01 kernel: md: using maximum available idle IO bandwidth (but
 not more than 200000 KB/sec) for recovery.
Dec 18 13:55:01 bp-01 kernel: md: using 128k window, over a total of 104857600k.
Dec 18 13:55:02 bp-01 kernel: md/raid1:mdX: active with 2 out of 3 mirrors
Dec 18 13:55:02 bp-01 kernel: created bitmap (100 pages) for device mdX
Dec 18 13:55:03 bp-01 kernel: md: md_do_sync() got signal ... exiting
Dec 18 13:55:03 bp-01 kernel: mdX: bitmap initialized from disk: read 7 pages, s
et 201692 of 204800 bits
Dec 18 13:55:03 bp-01 lvm[10557]: Monitoring RAID device vg-lv for events.
Dec 18 13:55:03 bp-01 kernel: md: recovery of RAID array mdX
Dec 18 13:55:03 bp-01 kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Dec 18 13:55:03 bp-01 lvm[10557]: Faulty devices in vg/lv successfully replaced.
Comment 19 Jonathan Earl Brassow 2012-12-18 20:42:56 UTC 
commit 970dfbcd691addb7b69f4011c067eba3b35270b6
Author: Jonathan Brassow <jbrassow>
Date:   Tue Dec 18 14:40:42 2012 -0600

    RAID:  Limit replacement of devices when array is not in-sync.
    
    If a RAID array is not in-sync, replacing devices should not be allowed
    as a general rule.  This is because the contents used to populate the
    incoming device may be undefined because the devices being read where
    not in-sync.  The kernel enforces this rule unless overridden by not
    allowing the creation of an array that is not in-sync and includes a
    devices that needs to be rebuilt.
    
    Since we cannot know the sync state of an LV if it is inactive, we must
    also enforce the rule that an array must be active to replace devices.
    
    That leaves us with the following conditions:
    1) never allow replacement or repair of devices if the LV is in-active
    2) never allow replacement if the LV is not in-sync
    3) allow repair if the LV is not in-sync, but warn that contents may
       not be recoverable.
    
    In the case where a user is performing the repair on the command line via
    'lvconvert --repair', the warning is printed before the user is prompted
    if they would like to replace the device(s).  If the repair is automated
    (i.e. via dmeventd and policy is "allocate"), then the device is replaced
    if possible and the warning is printed.
Comment 21 Corey Marthaler 2012-12-20 22:08:52 UTC 
Fix verified. I ran multiple iterations of the kill_random_non_synced_raid1_2legs with both the warn and allocate raid_fault_policy and saw no issues.

2.6.32-348.el6.x86_64
lvm2-2.02.98-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
lvm2-libs-2.02.98-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
lvm2-cluster-2.02.98-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
device-mapper-libs-1.02.77-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
device-mapper-event-1.02.77-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
device-mapper-event-libs-1.02.77-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
cmirror-2.02.98-6.el6    BUILT: Thu Dec 20 07:00:04 CST 2012
Comment 22 errata-xmlrpc 2013-02-21 08:03:36 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0501.html