Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 359341

Summary: Mirror corruption after one of three legs fail simultaneously on more than 1 mirror
Product: [Retired] Red Hat Cluster Suite Reporter: Corey Marthaler <cmarthal>
Component: lvm2-clusterAssignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED WONTFIX QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 4CC: agk, cfeist, dwysocha, jbrassow, mbroz, prockai
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-05-19 14:22:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 456575    
Attachments:
Description Flags
this is the non corrupt mirror (syncd_primary_3legs_1)
none
this is the corrupt mirror (syncd_primary_3legs_2)
none
updated test allows specification of devices.
none
Updated 'diff_test.sh' none

Description Corey Marthaler 2007-10-30 22:19:06 UTC 
Description of problem:
I've seen this twice now where there's i/o corruption on one of three mirrors
after one of the three legs (on each of those 3 mirrors) fail. This is on a 6
node cluster.

[root@grant-01 ~]# cman_tool nodes
Node  Votes Exp Sts  Name
   1    1    6   M   grant-02
   2    1    6   M   link-02
   3    1    6   M   grant-01
   4    1    6   M   grant-03
   5    1    6   M   link-08
   6    1    6   M   link-07


================================================================================
Iteration 2.2 started at Tue Oct 30 16:54:39 CDT 2007
================================================================================
Senario: Kill primary leg of synced 3 leg mirror(s)

****** Mirror hash info for this scenario ******
* name:      syncd_primary_3legs
* sync:      1
* mirrors:   3
* disklog:   1
* failpv:    /dev/sdb1
* legs:      3
* pvs:       /dev/sdb1 /dev/sdf1 /dev/sdg1 /dev/sdc1
************************************************

Creating mirror(s) on link-08...
qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_1 -L 800M helter_skelter
/dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150
qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_2 -L 800M helter_skelter
/dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150
qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_3 -L 800M helter_skelter
/dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150

Waiting for mirror(s) to sync
Verifying fully syncd mirror(s), currently at
 ...0.50% ...7.00% ...13.50% ...19.50% ...25.50% ...31.50% ...37.50% ...43.50%
...49.50% ...56.00% ...62.50% ...68.50% ...75.00% ...82.50% ...90.50% ...98.00%
...100.00%

Creating gfs on top of mirror(s) on link-08...
Mounting mirrored gfs filesystems on link-02...
Mounting mirrored gfs filesystems on link-07...
Mounting mirrored gfs filesystems on link-08...
Mounting mirrored gfs filesystems on grant-01...
Mounting mirrored gfs filesystems on grant-02...
Mounting mirrored gfs filesystems on grant-03...

Writing verification files (checkit) to mirror(s) on...
        link-02:
checkit starting with:
CREATE
Num files:          100
Random Seed:        20565
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        20569
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        20573
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
        link-07:
checkit starting with:
CREATE
Num files:          100
Random Seed:        18549
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        18553
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        18557
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
        link-08:
checkit starting with:
CREATE
Num files:          100
Random Seed:        18733
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        18737
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        18741
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
        grant-01:
checkit starting with:
CREATE
Num files:          100
Random Seed:        17985
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17989
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17993
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
        grant-02:
checkit starting with:
CREATE
Num files:          100
Random Seed:        17179
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17183
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17187
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
        grant-03:
checkit starting with:
CREATE
Num files:          100
Random Seed:        17963
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17967
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
CREATE
Num files:          100
Random Seed:        17971
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit

### NOT!! ### Starting the io load (collie/xdoio) on mirror(s)
Sleeping 15 seconds to get some I/O locks outstanding before the failure

Disabling device sdb on link-02
Disabling device sdb on link-07
Disabling device sdb on link-08
Disabling device sdc on grant-01
Disabling device sdc on grant-02
Disabling device sdc on grant-03

Attempting I/O to cause mirror down conversion(s) on link-08
10+0 records in
10+0 records out
Verifying the down conversion of the failed mirror(s)
  /dev/sdb1: read failed after 0 of 2048 at 0: Input/output error
Verifying FAILED device /dev/sdb1 is *NOT* in the volume(s)
  /dev/sdb1: open failed: No such device or address
Verifying LEG device /dev/sdf1 *IS* in the volume(s)
  /dev/sdb1: open failed: No such device or address
Verifying LEG device /dev/sdg1 *IS* in the volume(s)
  /dev/sdb1: open failed: No such device or address
Verifying LOG device /dev/sdc1 *IS* in the mirror(s)
  /dev/sdb1: open failed: No such device or address

Verifying files (checkit) on mirror(s) on...
        link-02
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1
Working dir:        /mnt/syncd_primary_3legs_1/checkit
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2
Working dir:        /mnt/syncd_primary_3legs_2/checkit
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3
Working dir:        /mnt/syncd_primary_3legs_3/checkit
Can not stat ajmwsa: Input/output error
checkit write verify failed
run_iteration() method failed at
/home/msp/cmarthal/work/rhel4/sts-root/lib/FI_engine.pm line 21.


Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: fatal: invalid
metadata block
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0:   bh = 165896 (type:
exp=6, found=0)
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0:   function = get_leaf
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0:   file =
/root/cluster-rhel46/gfs-kernel/src/gfs/dir.c, line = 438
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0:   time = 1193781138
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: about to withdraw
from the cluster
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: waiting for
outstanding I/O
Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: telling LM to withdraw
lock_dlm: withdraw abandoned memory
GFS: fsid=LINK_128:gfs3.0: withdrawn
Oct 30 16:52:20 link-02 kernel: lock_dlm: withdraw abandoned memory
Oct 30 16:52:20 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: withdrawn

According to 'dmsetup table' all nodes think that the server is node 1.

LINK-02:
Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO:
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   log type    : disk
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   in_sync     : YES
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   suspended   : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_id   : 1
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_valid: YES
Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO:
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   log type    : disk
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   in_sync     : YES
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   suspended   : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_id   : 1
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_valid: YES
Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO:
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   log type    : disk
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 16:56:45 link-02 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   in_sync     : YES
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   suspended   : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_id   : 1
Oct 30 16:56:45 link-02 kernel: dm-cmirror:   server_valid: YES



LINK-07:
Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO:
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   log type    : disk
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_valid: YES
Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO:
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   log type    : disk
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_valid: YES
Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO:
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   log type    : disk
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:20:30 link-07 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:20:30 link-07 kernel: dm-cmirror:   server_valid: YES



LINK-08:
Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO:
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   log type    : disk
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_valid: YES
Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO:
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   log type    : disk
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_valid: YES
Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO:
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid:
LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   uuid_ref    : 2
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   log type    : disk
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?region_count: 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_count  : 1600
Oct 30 17:19:01 link-08 kernel: dm-cmirror:  ?sync_search : 0
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   in_sync     : YES
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   suspended   : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   recovery_halted : NO
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_id   : 1
Oct 30 17:19:01 link-08 kernel: dm-cmirror:   server_valid: YES


Version-Release number of selected component (if applicable):
2.6.9-65.BRsmp  (built by Jon with fix for bz 290821)
cmirror-1.0.1-1
cmirror-kernel-2.6.9-38.3
lvm2-2.02.27-2.el4
lvm2-cluster-2.02.27-2.el4


More info to come...
Comment 1 Corey Marthaler 2007-10-30 22:22:08 UTC 
[root@link-02 tmp]# touch /mnt/syncd_primary_3legs_2/foo
[root@link-02 tmp]# touch /mnt/syncd_primary_3legs_1/foo
[root@link-02 tmp]# touch /mnt/syncd_primary_3legs_3/foo
touch: cannot touch `/mnt/syncd_primary_3legs_3/foo': Input/output error


[root@link-02 tmp]# lvs -a -o +devices
  /dev/sdb1: open failed: No such device or address
  LV                               VG             Attr   LSize   Origin Snap% 
Move Log                        Copy%  Devices
  LogVol00                         VolGroup00     -wi-ao  35.16G               
                                      /dev/hda2(0)
  LogVol01                         VolGroup00     -wi-ao   1.94G               
                                      /dev/hda2(1125)
  syncd_primary_3legs_1            helter_skelter mwi-ao 800.00M               
    syncd_primary_3legs_1_mlog 100.00
syncd_primary_3legs_1_mimage_2(0),syncd_primary_3legs_1_mimage_1(0)
  [syncd_primary_3legs_1_mimage_1] helter_skelter iwi-ao 800.00M               
                                      /dev/sdf1(0)
  [syncd_primary_3legs_1_mimage_2] helter_skelter iwi-ao 800.00M               
                                      /dev/sdg1(0)
  [syncd_primary_3legs_1_mlog]     helter_skelter lwi-ao   4.00M               
                                      /dev/sdc1(0)
  syncd_primary_3legs_2            helter_skelter mwi-ao 800.00M               
    syncd_primary_3legs_2_mlog 100.00
syncd_primary_3legs_2_mimage_2(0),syncd_primary_3legs_2_mimage_1(0)
  [syncd_primary_3legs_2_mimage_1] helter_skelter iwi-ao 800.00M               
                                      /dev/sdf1(200)
  [syncd_primary_3legs_2_mimage_2] helter_skelter iwi-ao 800.00M               
                                      /dev/sdg1(200)
  [syncd_primary_3legs_2_mlog]     helter_skelter lwi-ao   4.00M               
                                      /dev/sdc1(1)
  syncd_primary_3legs_3            helter_skelter mwi-ao 800.00M               
    syncd_primary_3legs_3_mlog 100.00
syncd_primary_3legs_3_mimage_2(0),syncd_primary_3legs_3_mimage_1(0)
  [syncd_primary_3legs_3_mimage_1] helter_skelter iwi-ao 800.00M               
                                      /dev/sdf1(400)
  [syncd_primary_3legs_3_mimage_2] helter_skelter iwi-ao 800.00M               
                                      /dev/sdg1(400)
  [syncd_primary_3legs_3_mlog]     helter_skelter lwi-ao   4.00M               
                                      /dev/sdc1(2)




[root@link-02 tmp]# dmsetup ls --tree
helter_skelter-syncd_primary_3legs_3 (253:16)
 ├─helter_skelter-syncd_primary_3legs_3_mimage_1 (253:14)
 │  └─ (8:81)
 ├─helter_skelter-syncd_primary_3legs_3_mimage_2 (253:15)
 │  └─ (8:97)
 └─helter_skelter-syncd_primary_3legs_3_mlog (253:12)
    └─ (8:33)
helter_skelter-syncd_primary_3legs_2 (253:11)
 ├─helter_skelter-syncd_primary_3legs_2_mimage_1 (253:9)
 │  └─ (8:81)
 ├─helter_skelter-syncd_primary_3legs_2_mimage_2 (253:10)
 │  └─ (8:97)
 └─helter_skelter-syncd_primary_3legs_2_mlog (253:7)
    └─ (8:33)
helter_skelter-syncd_primary_3legs_1 (253:6)
 ├─helter_skelter-syncd_primary_3legs_1_mimage_1 (253:4)
 │  └─ (8:81)
 ├─helter_skelter-syncd_primary_3legs_1_mimage_2 (253:5)
 │  └─ (8:97)
 └─helter_skelter-syncd_primary_3legs_1_mlog (253:2)
    └─ (8:33)
VolGroup00-LogVol01 (253:1)
 └─ (3:2)
VolGroup00-LogVol00 (253:0)
 └─ (3:2)
[root@link-02 tmp]# dmsetup status
helter_skelter-syncd_primary_3legs_3: 0 1638400 mirror 2 253:15 253:14 1600/1600
1 AA 3 clustered_disk 253:12 A
helter_skelter-syncd_primary_3legs_2: 0 1638400 mirror 2 253:10 253:9 1600/1600
1 AA 3 clustered_disk 253:7 A
helter_skelter-syncd_primary_3legs_1_mlog: 0 8192 linear
helter_skelter-syncd_primary_3legs_1: 0 1638400 mirror 2 253:5 253:4 1600/1600 1
AA 3 clustered_disk 253:2 A
helter_skelter-syncd_primary_3legs_3_mimage_2: 0 1638400 linear
helter_skelter-syncd_primary_3legs_2_mlog: 0 8192 linear
helter_skelter-syncd_primary_3legs_3_mimage_1: 0 1638400 linear
VolGroup00-LogVol01: 0 4063232 linear
helter_skelter-syncd_primary_3legs_2_mimage_2: 0 1638400 linear
helter_skelter-syncd_primary_3legs_3_mlog: 0 8192 linear
VolGroup00-LogVol00: 0 73728000 linear
helter_skelter-syncd_primary_3legs_2_mimage_1: 0 1638400 linear
helter_skelter-syncd_primary_3legs_1_mimage_2: 0 1638400 linear
helter_skelter-syncd_primary_3legs_1_mimage_1: 0 1638400 linear
[root@link-02 tmp]# dmsetup info
Name:              helter_skelter-syncd_primary_3legs_3
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      3
Major, minor:      253, 16
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1aMdW08e03Dk4PDguqHlwa8qPvOOpbrrf

Name:              helter_skelter-syncd_primary_3legs_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      17
Major, minor:      253, 11
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1i6hcIFu1KX4KIbCJdiqZm2G4IF7U4sQx

Name:              helter_skelter-syncd_primary_3legs_1_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 2
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5

Name:              helter_skelter-syncd_primary_3legs_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      9
Major, minor:      253, 6
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LgIaUSqsxMfpONXde3fOEseId1nREUd4

Name:              helter_skelter-syncd_primary_3legs_3_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 15
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1O8chBelt8Qri8ZRbJFchjmQq6yPKVe5B

Name:              helter_skelter-syncd_primary_3legs_2_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 7
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75

Name:              helter_skelter-syncd_primary_3legs_3_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 14
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1BU93JId0Ce5G7dacpRswQT1NEY8gy13l

Name:              VolGroup00-LogVol01
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: LVM-0mpjJpcho7U2NPluVVHwoJpLj1Z6a2VcYYhWLYsKSE0wcbNdieVGH93CfCLFc0k2

Name:              helter_skelter-syncd_primary_3legs_2_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 10
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1at8e02xYq0z3ZKbgbB4EBL01qgqReUcJ

Name:              helter_skelter-syncd_primary_3legs_3_mlog
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 12
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk

Name:              VolGroup00-LogVol00
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-0mpjJpcho7U2NPluVVHwoJpLj1Z6a2VcVVEtseAZ98h5LgRR3eX7h8BC4SRKAyfs

Name:              helter_skelter-syncd_primary_3legs_2_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 9
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sUFygkW2ogRWo4wuin62iV3lcmDi280A

Name:              helter_skelter-syncd_primary_3legs_1_mimage_2
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 5
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1Nxvaudl9tvy9Z1dWUTXUwFAdNq7LlLBV

Name:              helter_skelter-syncd_primary_3legs_1_mimage_1
State:             ACTIVE
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 4
Number of targets: 1
UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1do0EEz5v04hkupMhgHmA0xxN9XaSX83e
Comment 2 Corey Marthaler 2007-10-30 22:34:50 UTC 
[root@link-02 tmp]# ./find_dm_diffs.pl /dev/helter_skelter/syncd_primary_3legs_3
Found helter_skelter-syncd_primary_3legs_3_mimage_2
Found helter_skelter-syncd_primary_3legs_3_mimage_1
Suspending helter_skelter/syncd_primary_3legs_3 for now...
Device is 1638400 blocks
Diffing helter_skelter-syncd_primary_3legs_3_mimage_1 and
helter_skelter-syncd_primary_3legs_3_mimage_2
      0 -  262143: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
 262144 -  524287: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
 524288 -  786431: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
 786432 -  790527: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 790528 -  794623: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 794624 -  798719: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 798720 -  802815: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 802816 -  806911: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 806912 -  811007: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 811008 -  815103: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 815104 -  819199: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 819200 -  823295: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 823296 -  827391: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 827392 -  831487: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 831488 -  835583: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 835584 -  839679: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 839680 -  843775: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 843776 -  847871: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 847872 -  851967: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 851968 -  856063: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 856064 -  860159: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 860160 -  864255: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 864256 -  868351: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 868352 -  872447: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 872448 -  876543: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 876544 -  880639: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 880640 -  884735: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 884736 -  888831: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 888832 -  892927: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 892928 -  897023: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 897024 -  901119: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 901120 -  905215: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 905216 -  909311: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 909312 -  913407: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 913408 -  917503: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 917504 -  921599: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 921600 -  925695: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 925696 -  929791: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 929792 -  933887: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 933888 -  937983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 937984 -  942079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 942080 -  946175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 946176 -  950271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 950272 -  954367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 954368 -  958463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 958464 -  962559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 962560 -  966655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 966656 -  970751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 970752 -  974847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 974848 -  978943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 978944 -  983039: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 983040 -  987135: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 987136 -  991231: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 991232 -  995327: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 995328 -  999423: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
 999424 - 1003519: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1003520 - 1007615: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1007616 - 1011711: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1011712 - 1015807: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1015808 - 1015871: ................bbbbbbbbbbbb...bbbb.bb..bbb.....bbb.....bbb.bb.b
1015872 - 1015935: bbbbbbbbbbbbbbbbbbbbb.bbbbbbbbbbbbbb....bbbbbbbbbbb.....bbbbbbbb
1015936 - 1015999: bbb...bbb.b.....b.......bb......b...bbbbbbbbbbbbbbb.....bbb.....
1016000 - 1016063: bbb.....bbb.....bb......bb......bb......b.......bb......bbb.....
1016064 - 1016127: bb......bb......bbbbbbbbbbbbbbbbbbbbbbbbbb...bbbbb...bb.bbb....b
1016128 - 1016191: bbbbb...bbb.b...bbb..bb.bb......bbb..bbbbb......bbb.............
1016192 - 1016255: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.bbbbbbbbbbb..bbb.
1016256 - 1016319: bbbbb.bbbbbbbbbbbb.bbbbbbbbbbbbbbbb....bbbb.....bbbb....bbbbbbbb
1016320 - 1016383: bbbbbbbbbbbbbbbb................................................
1016384 - 1016447: ................................................................
1016448 - 1020543: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1020544 - 1024639: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1024640 - 1028735: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1028736 - 1032831: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1032832 - 1036927: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1036928 - 1041023: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1041024 - 1045119: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1045120 - 1049215: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1049216 - 1053311: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1053312 - 1057407: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1057408 - 1061503: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1061504 - 1065599: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1065600 - 1069695: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1069696 - 1073791: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1073792 - 1077887: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1077888 - 1081983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1081984 - 1086079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1086080 - 1090175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1090176 - 1094271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1094272 - 1098367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1098368 - 1102463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1102464 - 1106559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1106560 - 1110655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1110656 - 1114751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1114752 - 1118847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1118848 - 1122943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1122944 - 1127039: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1127040 - 1131135: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1131136 - 1135231: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1135232 - 1139327: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1139328 - 1143423: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1143424 - 1147519: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1147520 - 1151615: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1151616 - 1155711: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1155712 - 1159807: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1159808 - 1163903: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1163904 - 1167999: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1168000 - 1168063: ................................................................
1168064 - 1168127: ................................................................
1168128 - 1168191: ................................................................
1168192 - 1168255: ................................................................
1168256 - 1168319: ................................................................
1168320 - 1168383: ................................................................
1168384 - 1168447: ................................................................
1168448 - 1168511: ................................................................
1168512 - 1168575: ................................................................
1168576 - 1168639: ................................................................
1168640 - 1168703: ................................................................
1168704 - 1168767: ................................................................
1168768 - 1168831: ................................................................
1168832 - 1168895: ................................................................
1168896 - 1168959: ................................................................
1168960 - 1169023: ................................................................
1169024 - 1169087: ................................................................
1169088 - 1169151: ................................................................
1169152 - 1169215: ................................................................
1169216 - 1169279: ................................................................
1169280 - 1169343: ................................................................
1169344 - 1169407: ................................................................
1169408 - 1169471: ................................................................
1169472 - 1169535: ................................................................
1169536 - 1169599: ................................................................
1169600 - 1169663: ................................................................
1169664 - 1169727: ................................................................
1169728 - 1169791: ................................................................
1169792 - 1169855: ................................................................
1169856 - 1169919: ................................................................
1169920 - 1169983: ................................................................
1169984 - 1170047: ................................................................
1170048 - 1170111: ................................................................
1170112 - 1170175: ................................................................
1170176 - 1170239: ................................................................
1170240 - 1170303: ................................................................
1170304 - 1170367: ................................................................
1170368 - 1170431: ................................................................
1170432 - 1170495: ................................................................
1170496 - 1170559: ................................................................
1170560 - 1170623: ................................................................
1170624 - 1170687: ................................................................
1170688 - 1170751: ................................................................
1170752 - 1170815: ................................................................
1170816 - 1170879: ................................................................
1170880 - 1170943: ................................................................
1170944 - 1171007: ................................................................
1171008 - 1171071: ................................................................
1171072 - 1171135: ................................................................
1171136 - 1171199: ................................................................
1171200 - 1171263: ................................................................
1171264 - 1171327: ................................................................
1171328 - 1171391: ................................................................
1171392 - 1171455: ................................................................
1171456 - 1171519: ................bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1171520 - 1171583: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1171584 - 1171647: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1171648 - 1171711: bb......bb......bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1171712 - 1171775: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.......bbb.....bbb.....bb......
1171776 - 1171839: bbbbbbbbbbbbbbbbbbbbbbbbbb...bbbbb......bbbbbbbbbbbbbb.bbbbbbbbb
1171840 - 1171903: bbbbb..bbb....b.bb.bbbbbbbbbbbbbbbbbb...bb......bbbbbb..bbbbbbbb
1171904 - 1171967: bbbbbbbbb......bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1171968 - 1172031: bbbbbbbbbbbbbbbb................................................
1172032 - 1172095: ................................................................
1172096 - 1176191: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1176192 - 1180287: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1180288 - 1184383: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1184384 - 1188479: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1188480 - 1192575: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1192576 - 1196671: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1196672 - 1200767: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1200768 - 1204863: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1204864 - 1208959: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1208960 - 1213055: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1213056 - 1217151: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1217152 - 1221247: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1221248 - 1225343: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1225344 - 1229439: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1229440 - 1233535: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1233536 - 1237631: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1237632 - 1241727: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1241728 - 1245823: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1245824 - 1249919: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1249920 - 1254015: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1254016 - 1258111: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1258112 - 1262207: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1262208 - 1266303: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1266304 - 1270399: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1270400 - 1274495: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1274496 - 1278591: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1278592 - 1282687: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1282688 - 1286783: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1286784 - 1290879: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1290880 - 1294975: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1294976 - 1299071: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1299072 - 1303167: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1303168 - 1307263: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1307264 - 1311359: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1311360 - 1315455: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1315456 - 1319551: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1319552 - 1323647: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1323648 - 1323711: ................................................................
1323712 - 1323775: ................................................................
1323776 - 1323839: ................................................................
1323840 - 1323903: ................................................................
1323904 - 1323967: ................................................................
1323968 - 1324031: ................................................................
1324032 - 1324095: ................................................................
1324096 - 1324159: ................................................................
1324160 - 1324223: ................................................................
1324224 - 1324287: ................................................................
1324288 - 1324351: ................................................................
1324352 - 1324415: ................................................................
1324416 - 1324479: ................................................................
1324480 - 1324543: ................................................................
1324544 - 1324607: ................................................................
1324608 - 1324671: ................................................................
1324672 - 1324735: ................................................................
1324736 - 1324799: ................................................................
1324800 - 1324863: ................................................................
1324864 - 1324927: ................................................................
1324928 - 1324991: ................................................................
1324992 - 1325055: ................................................................
1325056 - 1325119: ................................................................
1325120 - 1325183: ................................................................
1325184 - 1325247: ................................................................
1325248 - 1325311: ................................................................
1325312 - 1325375: ................................................................
1325376 - 1325439: ................................................................
1325440 - 1325503: ................................................................
1325504 - 1325567: ................................................................
1325568 - 1325631: ................................................................
1325632 - 1325695: ................................................................
1325696 - 1325759: ................................................................
1325760 - 1325823: ................................................................
1325824 - 1325887: ................................................................
1325888 - 1325951: ................................................................
1325952 - 1326015: ................................................................
1326016 - 1326079: ................................................................
1326080 - 1326143: ................................................................
1326144 - 1326207: ................................................................
1326208 - 1326271: ................................................................
1326272 - 1326335: ................................................................
1326336 - 1326399: ................................................................
1326400 - 1326463: ................................................................
1326464 - 1326527: ................................................................
1326528 - 1326591: ................................................................
1326592 - 1326655: ................................................................
1326656 - 1326719: ................................................................
1326720 - 1326783: ................................................................
1326784 - 1326847: ................................................................
1326848 - 1326911: ................................................................
1326912 - 1326975: ................................................................
1326976 - 1327039: ................................................................
1327040 - 1327103: ................................................................
1327104 - 1327167: ................bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327168 - 1327231: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327232 - 1327295: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327296 - 1327359: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327360 - 1327423: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327424 - 1327487: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.....bbbbbbbbbbbbbbbbbbbbbbbbb
1327488 - 1327551: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.....bbb.....bbbb....bbb.....
1327552 - 1327615: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
1327616 - 1327679: bbbbbbbbbbbbbbbb................................................
1327680 - 1327743: ................................................................
1327744 - 1589887: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
1589888 - 1593983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1593984 - 1598079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1598080 - 1602175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1602176 - 1606271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1606272 - 1610367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1610368 - 1614463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1614464 - 1618559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1618560 - 1622655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1622656 - 1626751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1626752 - 1630847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1630848 - 1634943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
1634944 - 1635007: ................................................................
1635008 - 1635071: ................................................................
1635072 - 1635135: ................................................................
1635136 - 1635199: ................................................................
1635200 - 1635263: ................................................................
1635264 - 1635327: ................................................................
1635328 - 1635391: ................................................................
1635392 - 1635455: ................................................................
1635456 - 1635519: ................................................................
1635520 - 1635583: ................................................................
1635584 - 1635647: ................................................................
1635648 - 1635711: ................................................................
1635712 - 1635775: ................................................................
1635776 - 1635839: ................................................................
1635840 - 1635903: ................................................................
1635904 - 1635967: ................................................................
1635968 - 1636031: ................................................................
1636032 - 1636095: ................................................................
1636096 - 1636159: ................................................................
1636160 - 1636223: ................................................................
1636224 - 1636287: ................................................................
1636288 - 1636351: ................................................................
1636352 - 1636415: ................................................................
1636416 - 1636479: ................................................................
1636480 - 1636543: ................................................................
1636544 - 1636607: ................................................................
1636608 - 1636671: ................................................................
1636672 - 1636735: ................................................................
1636736 - 1636799: ................................................................
1636800 - 1636863: ................................................................
1636864 - 1636927: ................................................................
1636928 - 1636991: ................................................................
1636992 - 1637055: ................................................................
1637056 - 1637119: ................................................................
1637120 - 1637183: ................................................................
1637184 - 1637247: ................................................................
1637248 - 1637311: ................................................................
1637312 - 1637375: ................................................................
1637376 - 1637439: ................................................................
1637440 - 1637503: ................................................................
1637504 - 1637567: ................................................................
1637568 - 1637631: ................................................................
1637632 - 1637695: ................................................................
1637696 - 1637759: ................................................................
1637760 - 1637823: ................................................................
1637824 - 1637887: ................................................................
1637888 - 1637951: ................................................................
1637952 - 1638015: ................................................................
1638016 - 1638079: ................................................................
1638080 - 1638143: ................................................................
1638144 - 1638207: ................................................................
1638208 - 1638271: ................................................................
1638272 - 1638335: ................................................................
1638336 - 1638399: ................................................................

Resuming helter_skelter/syncd_primary_3legs_3
1254 blocks differ
0.0765380859375% of the mirror is bad.
Comment 3 Corey Marthaler 2007-11-01 13:58:45 UTC 
Reproduced this again last night with only two mirrors (but again both with
three legs). This test now verifies that all legs are in sync (instead of just
the last one created) before doing any leg failure operations. Again, it's the
last mirror which becomes corrupt.

[root@link-02 ~]# lvs -a -o +devices
  /dev/sdg1: open failed: No such device or address
  LV                               VG             Attr   LSize   Origin Snap% 
Move Log                        Copy%  Devices                     
  LogVol00                         VolGroup00     -wi-ao  35.16G               
                                      /dev/hda2(0)                
  LogVol01                         VolGroup00     -wi-ao   1.94G               
                                      /dev/hda2(1125)             
  syncd_primary_3legs_1            helter_skelter mwi-ao 800.00M               
    syncd_primary_3legs_1_mlog 100.00
syncd_primary_3legs_1_mimage_2(0),syncd_primary_3legs_1_mimage_1(0)
  [syncd_primary_3legs_1_mimage_1] helter_skelter iwi-ao 800.00M               
                                      /dev/sdc1(0)                
  [syncd_primary_3legs_1_mimage_2] helter_skelter iwi-ao 800.00M               
                                      /dev/sdd1(0)                
  [syncd_primary_3legs_1_mlog]     helter_skelter lwi-ao   4.00M               
                                      /dev/sde1(0)                
  syncd_primary_3legs_2            helter_skelter mwi-ao 800.00M               
    syncd_primary_3legs_2_mlog 100.00
syncd_primary_3legs_2_mimage_2(0),syncd_primary_3legs_2_mimage_1(0)
  [syncd_primary_3legs_2_mimage_1] helter_skelter iwi-ao 800.00M               
                                      /dev/sdc1(200)              
  [syncd_primary_3legs_2_mimage_2] helter_skelter iwi-ao 800.00M               
                                      /dev/sdd1(200)              
  [syncd_primary_3legs_2_mlog]     helter_skelter lwi-ao   4.00M               
                                      /dev/sde1(1)                

[root@link-02 ~]# touch /mnt/syncd_primary_3legs_1/foo
[root@link-02 ~]# touch /mnt/syncd_primary_3legs_2/foo
touch: cannot touch `/mnt/syncd_primary_3legs_2/foo': Input/output error

I'll run 'find_dm_diffs' on both the good mirror and the bad mirror and attach
the results.
Comment 4 Corey Marthaler 2007-11-01 14:19:21 UTC 
Created attachment 245571 [details]
this is the non corrupt mirror (syncd_primary_3legs_1)
Comment 5 Corey Marthaler 2007-11-01 14:19:52 UTC 
Created attachment 245581 [details]
this is the corrupt mirror (syncd_primary_3legs_2)
Comment 6 Corey Marthaler 2007-11-01 18:24:27 UTC 
Jon,

Should this be a candidate for a 4.6.Z errata like 290821?
Comment 7 Jonathan Earl Brassow 2007-11-02 04:05:16 UTC 
Any bug that has been shown to corrupt data certainly gets first priority.

Let's try to limit the scope.
1) Can you make this happen with just one mirror?
2) Can you make this happen with two-way mirrors?  (You won't be able to use the
checker script, but GFS will obviously still complain.)
3) Are the servers for the mirrors located on the same or different machines? 
(dmsetup table ; <view> /var/log/messages)  Is there a pattern that can be
linked to the last mirror, as that seems to be the one with the problem?
Comment 8 Corey Marthaler 2007-11-02 21:47:11 UTC 
> 1) Can you make this happen with just one mirror?

No, it appears that you need at least 2

> 2) Can you make this happen with two-way mirrors?

No, it appears that you need 3 legs

> 3) Are the servers for the mirrors located on the same or different machines? 

I've seen it where the servers were on different nodes and where they were on
the same node
Comment 10 Jonathan Earl Brassow 2007-11-13 18:36:19 UTC 
Created attachment 257231 [details]
updated test allows specification of devices.

Able to reproduce with this new script using the following args:
$> diff_test.sh -m 2 -f 3 -S -d /dev/sda,/dev/sdb,/dev/sdc,/dev/sdd:0-50

Once I/O starts, I kill the primary
Comment 11 Jonathan Earl Brassow 2007-11-20 16:55:23 UTC 
reproducible with single machine mirroring
Comment 13 Jonathan Earl Brassow 2008-03-26 17:51:43 UTC 
*** Bug 243013 has been marked as a duplicate of this bug. ***
Comment 14 Jonathan Earl Brassow 2008-04-02 16:04:48 UTC 
Created attachment 300084 [details]
Updated 'diff_test.sh'

Updated test
Comment 18 Jonathan Earl Brassow 2010-05-19 14:22:01 UTC 
3-way mirrors are not supported on RHEL4.  Closing WONTFIX.  Please try to reproduce on later releases where 3-way mirrors are supported.