Description of problem: I've seen this twice now where there's i/o corruption on one of three mirrors after one of the three legs (on each of those 3 mirrors) fail. This is on a 6 node cluster. [root@grant-01 ~]# cman_tool nodes Node Votes Exp Sts Name 1 1 6 M grant-02 2 1 6 M link-02 3 1 6 M grant-01 4 1 6 M grant-03 5 1 6 M link-08 6 1 6 M link-07 ================================================================================ Iteration 2.2 started at Tue Oct 30 16:54:39 CDT 2007 ================================================================================ Senario: Kill primary leg of synced 3 leg mirror(s) ****** Mirror hash info for this scenario ****** * name: syncd_primary_3legs * sync: 1 * mirrors: 3 * disklog: 1 * failpv: /dev/sdb1 * legs: 3 * pvs: /dev/sdb1 /dev/sdf1 /dev/sdg1 /dev/sdc1 ************************************************ Creating mirror(s) on link-08... qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_1 -L 800M helter_skelter /dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150 qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_2 -L 800M helter_skelter /dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150 qarsh root@link-08 lvcreate -m 2 -n syncd_primary_3legs_3 -L 800M helter_skelter /dev/sdb1:0-1000 /dev/sdf1:0-1000 /dev/sdg1:0-1000 /dev/sdc1:0-150 Waiting for mirror(s) to sync Verifying fully syncd mirror(s), currently at ...0.50% ...7.00% ...13.50% ...19.50% ...25.50% ...31.50% ...37.50% ...43.50% ...49.50% ...56.00% ...62.50% ...68.50% ...75.00% ...82.50% ...90.50% ...98.00% ...100.00% Creating gfs on top of mirror(s) on link-08... Mounting mirrored gfs filesystems on link-02... Mounting mirrored gfs filesystems on link-07... Mounting mirrored gfs filesystems on link-08... Mounting mirrored gfs filesystems on grant-01... Mounting mirrored gfs filesystems on grant-02... Mounting mirrored gfs filesystems on grant-03... Writing verification files (checkit) to mirror(s) on... link-02: checkit starting with: CREATE Num files: 100 Random Seed: 20565 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 20569 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 20573 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit link-07: checkit starting with: CREATE Num files: 100 Random Seed: 18549 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 18553 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 18557 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit link-08: checkit starting with: CREATE Num files: 100 Random Seed: 18733 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 18737 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 18741 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit grant-01: checkit starting with: CREATE Num files: 100 Random Seed: 17985 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17989 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17993 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit grant-02: checkit starting with: CREATE Num files: 100 Random Seed: 17179 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17183 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17187 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit grant-03: checkit starting with: CREATE Num files: 100 Random Seed: 17963 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17967 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: CREATE Num files: 100 Random Seed: 17971 Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit ### NOT!! ### Starting the io load (collie/xdoio) on mirror(s) Sleeping 15 seconds to get some I/O locks outstanding before the failure Disabling device sdb on link-02 Disabling device sdb on link-07 Disabling device sdb on link-08 Disabling device sdc on grant-01 Disabling device sdc on grant-02 Disabling device sdc on grant-03 Attempting I/O to cause mirror down conversion(s) on link-08 10+0 records in 10+0 records out Verifying the down conversion of the failed mirror(s) /dev/sdb1: read failed after 0 of 2048 at 0: Input/output error Verifying FAILED device /dev/sdb1 is *NOT* in the volume(s) /dev/sdb1: open failed: No such device or address Verifying LEG device /dev/sdf1 *IS* in the volume(s) /dev/sdb1: open failed: No such device or address Verifying LEG device /dev/sdg1 *IS* in the volume(s) /dev/sdb1: open failed: No such device or address Verifying LOG device /dev/sdc1 *IS* in the mirror(s) /dev/sdb1: open failed: No such device or address Verifying files (checkit) on mirror(s) on... link-02 checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_1 Working dir: /mnt/syncd_primary_3legs_1/checkit checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_2 Working dir: /mnt/syncd_primary_3legs_2/checkit checkit starting with: VERIFY Verify XIOR Stream: /tmp/checkit_syncd_primary_3legs_3 Working dir: /mnt/syncd_primary_3legs_3/checkit Can not stat ajmwsa: Input/output error checkit write verify failed run_iteration() method failed at /home/msp/cmarthal/work/rhel4/sts-root/lib/FI_engine.pm line 21. Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: fatal: invalid metadata block Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: bh = 165896 (type: exp=6, found=0) Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: function = get_leaf Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: file = /root/cluster-rhel46/gfs-kernel/src/gfs/dir.c, line = 438 Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: time = 1193781138 Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: about to withdraw from the cluster Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: waiting for outstanding I/O Oct 30 16:52:18 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: telling LM to withdraw lock_dlm: withdraw abandoned memory GFS: fsid=LINK_128:gfs3.0: withdrawn Oct 30 16:52:20 link-02 kernel: lock_dlm: withdraw abandoned memory Oct 30 16:52:20 link-02 kernel: GFS: fsid=LINK_128:gfs3.0: withdrawn According to 'dmsetup table' all nodes think that the server is node 1. LINK-02: Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO: Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid_ref : 2 Oct 30 16:56:45 link-02 kernel: dm-cmirror: log type : disk Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?region_count: 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_search : 0 Oct 30 16:56:45 link-02 kernel: dm-cmirror: in_sync : YES Oct 30 16:56:45 link-02 kernel: dm-cmirror: suspended : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: recovery_halted : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_id : 1 Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_valid: YES Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO: Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75 Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid_ref : 2 Oct 30 16:56:45 link-02 kernel: dm-cmirror: log type : disk Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?region_count: 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_search : 0 Oct 30 16:56:45 link-02 kernel: dm-cmirror: in_sync : YES Oct 30 16:56:45 link-02 kernel: dm-cmirror: suspended : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: recovery_halted : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_id : 1 Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_valid: YES Oct 30 16:56:45 link-02 kernel: dm-cmirror: LOG INFO: Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5 Oct 30 16:56:45 link-02 kernel: dm-cmirror: uuid_ref : 2 Oct 30 16:56:45 link-02 kernel: dm-cmirror: log type : disk Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?region_count: 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 16:56:45 link-02 kernel: dm-cmirror: ?sync_search : 0 Oct 30 16:56:45 link-02 kernel: dm-cmirror: in_sync : YES Oct 30 16:56:45 link-02 kernel: dm-cmirror: suspended : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: recovery_halted : NO Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_id : 1 Oct 30 16:56:45 link-02 kernel: dm-cmirror: server_valid: YES LINK-07: Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO: Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:20:30 link-07 kernel: dm-cmirror: log type : disk Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:20:30 link-07 kernel: dm-cmirror: in_sync : YES Oct 30 17:20:30 link-07 kernel: dm-cmirror: suspended : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_id : 1 Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_valid: YES Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO: Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75 Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:20:30 link-07 kernel: dm-cmirror: log type : disk Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:20:30 link-07 kernel: dm-cmirror: in_sync : YES Oct 30 17:20:30 link-07 kernel: dm-cmirror: suspended : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_id : 1 Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_valid: YES Oct 30 17:20:30 link-07 kernel: dm-cmirror: LOG INFO: Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5 Oct 30 17:20:30 link-07 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:20:30 link-07 kernel: dm-cmirror: log type : disk Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:20:30 link-07 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:20:30 link-07 kernel: dm-cmirror: in_sync : YES Oct 30 17:20:30 link-07 kernel: dm-cmirror: suspended : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_id : 1 Oct 30 17:20:30 link-07 kernel: dm-cmirror: server_valid: YES LINK-08: Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO: Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:19:01 link-08 kernel: dm-cmirror: log type : disk Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:19:01 link-08 kernel: dm-cmirror: in_sync : YES Oct 30 17:19:01 link-08 kernel: dm-cmirror: suspended : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_id : 1 Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_valid: YES Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO: Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75 Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:19:01 link-08 kernel: dm-cmirror: log type : disk Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:19:01 link-08 kernel: dm-cmirror: in_sync : YES Oct 30 17:19:01 link-08 kernel: dm-cmirror: suspended : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_id : 1 Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_valid: YES Oct 30 17:19:01 link-08 kernel: dm-cmirror: LOG INFO: Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5 Oct 30 17:19:01 link-08 kernel: dm-cmirror: uuid_ref : 2 Oct 30 17:19:01 link-08 kernel: dm-cmirror: log type : disk Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?region_count: 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_count : 1600 Oct 30 17:19:01 link-08 kernel: dm-cmirror: ?sync_search : 0 Oct 30 17:19:01 link-08 kernel: dm-cmirror: in_sync : YES Oct 30 17:19:01 link-08 kernel: dm-cmirror: suspended : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: recovery_halted : NO Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_id : 1 Oct 30 17:19:01 link-08 kernel: dm-cmirror: server_valid: YES Version-Release number of selected component (if applicable): 2.6.9-65.BRsmp (built by Jon with fix for bz 290821) cmirror-1.0.1-1 cmirror-kernel-2.6.9-38.3 lvm2-2.02.27-2.el4 lvm2-cluster-2.02.27-2.el4 More info to come...
[root@link-02 tmp]# touch /mnt/syncd_primary_3legs_2/foo [root@link-02 tmp]# touch /mnt/syncd_primary_3legs_1/foo [root@link-02 tmp]# touch /mnt/syncd_primary_3legs_3/foo touch: cannot touch `/mnt/syncd_primary_3legs_3/foo': Input/output error [root@link-02 tmp]# lvs -a -o +devices /dev/sdb1: open failed: No such device or address LV VG Attr LSize Origin Snap% Move Log Copy% Devices LogVol00 VolGroup00 -wi-ao 35.16G /dev/hda2(0) LogVol01 VolGroup00 -wi-ao 1.94G /dev/hda2(1125) syncd_primary_3legs_1 helter_skelter mwi-ao 800.00M syncd_primary_3legs_1_mlog 100.00 syncd_primary_3legs_1_mimage_2(0),syncd_primary_3legs_1_mimage_1(0) [syncd_primary_3legs_1_mimage_1] helter_skelter iwi-ao 800.00M /dev/sdf1(0) [syncd_primary_3legs_1_mimage_2] helter_skelter iwi-ao 800.00M /dev/sdg1(0) [syncd_primary_3legs_1_mlog] helter_skelter lwi-ao 4.00M /dev/sdc1(0) syncd_primary_3legs_2 helter_skelter mwi-ao 800.00M syncd_primary_3legs_2_mlog 100.00 syncd_primary_3legs_2_mimage_2(0),syncd_primary_3legs_2_mimage_1(0) [syncd_primary_3legs_2_mimage_1] helter_skelter iwi-ao 800.00M /dev/sdf1(200) [syncd_primary_3legs_2_mimage_2] helter_skelter iwi-ao 800.00M /dev/sdg1(200) [syncd_primary_3legs_2_mlog] helter_skelter lwi-ao 4.00M /dev/sdc1(1) syncd_primary_3legs_3 helter_skelter mwi-ao 800.00M syncd_primary_3legs_3_mlog 100.00 syncd_primary_3legs_3_mimage_2(0),syncd_primary_3legs_3_mimage_1(0) [syncd_primary_3legs_3_mimage_1] helter_skelter iwi-ao 800.00M /dev/sdf1(400) [syncd_primary_3legs_3_mimage_2] helter_skelter iwi-ao 800.00M /dev/sdg1(400) [syncd_primary_3legs_3_mlog] helter_skelter lwi-ao 4.00M /dev/sdc1(2) [root@link-02 tmp]# dmsetup ls --tree helter_skelter-syncd_primary_3legs_3 (253:16) ├─helter_skelter-syncd_primary_3legs_3_mimage_1 (253:14) │ └─ (8:81) ├─helter_skelter-syncd_primary_3legs_3_mimage_2 (253:15) │ └─ (8:97) └─helter_skelter-syncd_primary_3legs_3_mlog (253:12) └─ (8:33) helter_skelter-syncd_primary_3legs_2 (253:11) ├─helter_skelter-syncd_primary_3legs_2_mimage_1 (253:9) │ └─ (8:81) ├─helter_skelter-syncd_primary_3legs_2_mimage_2 (253:10) │ └─ (8:97) └─helter_skelter-syncd_primary_3legs_2_mlog (253:7) └─ (8:33) helter_skelter-syncd_primary_3legs_1 (253:6) ├─helter_skelter-syncd_primary_3legs_1_mimage_1 (253:4) │ └─ (8:81) ├─helter_skelter-syncd_primary_3legs_1_mimage_2 (253:5) │ └─ (8:97) └─helter_skelter-syncd_primary_3legs_1_mlog (253:2) └─ (8:33) VolGroup00-LogVol01 (253:1) └─ (3:2) VolGroup00-LogVol00 (253:0) └─ (3:2) [root@link-02 tmp]# dmsetup status helter_skelter-syncd_primary_3legs_3: 0 1638400 mirror 2 253:15 253:14 1600/1600 1 AA 3 clustered_disk 253:12 A helter_skelter-syncd_primary_3legs_2: 0 1638400 mirror 2 253:10 253:9 1600/1600 1 AA 3 clustered_disk 253:7 A helter_skelter-syncd_primary_3legs_1_mlog: 0 8192 linear helter_skelter-syncd_primary_3legs_1: 0 1638400 mirror 2 253:5 253:4 1600/1600 1 AA 3 clustered_disk 253:2 A helter_skelter-syncd_primary_3legs_3_mimage_2: 0 1638400 linear helter_skelter-syncd_primary_3legs_2_mlog: 0 8192 linear helter_skelter-syncd_primary_3legs_3_mimage_1: 0 1638400 linear VolGroup00-LogVol01: 0 4063232 linear helter_skelter-syncd_primary_3legs_2_mimage_2: 0 1638400 linear helter_skelter-syncd_primary_3legs_3_mlog: 0 8192 linear VolGroup00-LogVol00: 0 73728000 linear helter_skelter-syncd_primary_3legs_2_mimage_1: 0 1638400 linear helter_skelter-syncd_primary_3legs_1_mimage_2: 0 1638400 linear helter_skelter-syncd_primary_3legs_1_mimage_1: 0 1638400 linear [root@link-02 tmp]# dmsetup info Name: helter_skelter-syncd_primary_3legs_3 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 3 Major, minor: 253, 16 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1aMdW08e03Dk4PDguqHlwa8qPvOOpbrrf Name: helter_skelter-syncd_primary_3legs_2 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 17 Major, minor: 253, 11 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1i6hcIFu1KX4KIbCJdiqZm2G4IF7U4sQx Name: helter_skelter-syncd_primary_3legs_1_mlog State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 2 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LRsUHZye8CMqO3uySeXsvaYbBkXyNKS5 Name: helter_skelter-syncd_primary_3legs_1 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 9 Major, minor: 253, 6 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1LgIaUSqsxMfpONXde3fOEseId1nREUd4 Name: helter_skelter-syncd_primary_3legs_3_mimage_2 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 15 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1O8chBelt8Qri8ZRbJFchjmQq6yPKVe5B Name: helter_skelter-syncd_primary_3legs_2_mlog State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 7 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1fJLwd5jbYdwgKMxSPDuUYTxwkaC1an75 Name: helter_skelter-syncd_primary_3legs_3_mimage_1 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 14 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1BU93JId0Ce5G7dacpRswQT1NEY8gy13l Name: VolGroup00-LogVol01 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 1 Number of targets: 1 UUID: LVM-0mpjJpcho7U2NPluVVHwoJpLj1Z6a2VcYYhWLYsKSE0wcbNdieVGH93CfCLFc0k2 Name: helter_skelter-syncd_primary_3legs_2_mimage_2 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 10 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1at8e02xYq0z3ZKbgbB4EBL01qgqReUcJ Name: helter_skelter-syncd_primary_3legs_3_mlog State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 12 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sA182ZPBv3JJcfVjZKEKdTYtqeB42bzk Name: VolGroup00-LogVol00 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 0 Number of targets: 1 UUID: LVM-0mpjJpcho7U2NPluVVHwoJpLj1Z6a2VcVVEtseAZ98h5LgRR3eX7h8BC4SRKAyfs Name: helter_skelter-syncd_primary_3legs_2_mimage_1 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 9 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1sUFygkW2ogRWo4wuin62iV3lcmDi280A Name: helter_skelter-syncd_primary_3legs_1_mimage_2 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 5 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1Nxvaudl9tvy9Z1dWUTXUwFAdNq7LlLBV Name: helter_skelter-syncd_primary_3legs_1_mimage_1 State: ACTIVE Tables present: LIVE Open count: 1 Event number: 0 Major, minor: 253, 4 Number of targets: 1 UUID: LVM-jkZvkjIDtKIfA7ByZs2y8pKBSbsMllX1do0EEz5v04hkupMhgHmA0xxN9XaSX83e
[root@link-02 tmp]# ./find_dm_diffs.pl /dev/helter_skelter/syncd_primary_3legs_3 Found helter_skelter-syncd_primary_3legs_3_mimage_2 Found helter_skelter-syncd_primary_3legs_3_mimage_1 Suspending helter_skelter/syncd_primary_3legs_3 for now... Device is 1638400 blocks Diffing helter_skelter-syncd_primary_3legs_3_mimage_1 and helter_skelter-syncd_primary_3legs_3_mimage_2 0 - 262143: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 262144 - 524287: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 524288 - 786431: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 786432 - 790527: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 790528 - 794623: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 794624 - 798719: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 798720 - 802815: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 802816 - 806911: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 806912 - 811007: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 811008 - 815103: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 815104 - 819199: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 819200 - 823295: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 823296 - 827391: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 827392 - 831487: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 831488 - 835583: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 835584 - 839679: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 839680 - 843775: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 843776 - 847871: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 847872 - 851967: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 851968 - 856063: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 856064 - 860159: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 860160 - 864255: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 864256 - 868351: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 868352 - 872447: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 872448 - 876543: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 876544 - 880639: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 880640 - 884735: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 884736 - 888831: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 888832 - 892927: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 892928 - 897023: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 897024 - 901119: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 901120 - 905215: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 905216 - 909311: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 909312 - 913407: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 913408 - 917503: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 917504 - 921599: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 921600 - 925695: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 925696 - 929791: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 929792 - 933887: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 933888 - 937983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 937984 - 942079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 942080 - 946175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 946176 - 950271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 950272 - 954367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 954368 - 958463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 958464 - 962559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 962560 - 966655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 966656 - 970751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 970752 - 974847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 974848 - 978943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 978944 - 983039: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 983040 - 987135: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 987136 - 991231: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 991232 - 995327: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 995328 - 999423: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 999424 - 1003519: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1003520 - 1007615: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1007616 - 1011711: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1011712 - 1015807: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1015808 - 1015871: ................bbbbbbbbbbbb...bbbb.bb..bbb.....bbb.....bbb.bb.b 1015872 - 1015935: bbbbbbbbbbbbbbbbbbbbb.bbbbbbbbbbbbbb....bbbbbbbbbbb.....bbbbbbbb 1015936 - 1015999: bbb...bbb.b.....b.......bb......b...bbbbbbbbbbbbbbb.....bbb..... 1016000 - 1016063: bbb.....bbb.....bb......bb......bb......b.......bb......bbb..... 1016064 - 1016127: bb......bb......bbbbbbbbbbbbbbbbbbbbbbbbbb...bbbbb...bb.bbb....b 1016128 - 1016191: bbbbb...bbb.b...bbb..bb.bb......bbb..bbbbb......bbb............. 1016192 - 1016255: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.bbbbbbbbbbb..bbb. 1016256 - 1016319: bbbbb.bbbbbbbbbbbb.bbbbbbbbbbbbbbbb....bbbb.....bbbb....bbbbbbbb 1016320 - 1016383: bbbbbbbbbbbbbbbb................................................ 1016384 - 1016447: ................................................................ 1016448 - 1020543: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1020544 - 1024639: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1024640 - 1028735: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1028736 - 1032831: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1032832 - 1036927: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1036928 - 1041023: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1041024 - 1045119: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1045120 - 1049215: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1049216 - 1053311: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1053312 - 1057407: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1057408 - 1061503: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1061504 - 1065599: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1065600 - 1069695: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1069696 - 1073791: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1073792 - 1077887: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1077888 - 1081983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1081984 - 1086079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1086080 - 1090175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1090176 - 1094271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1094272 - 1098367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1098368 - 1102463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1102464 - 1106559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1106560 - 1110655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1110656 - 1114751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1114752 - 1118847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1118848 - 1122943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1122944 - 1127039: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1127040 - 1131135: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1131136 - 1135231: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1135232 - 1139327: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1139328 - 1143423: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1143424 - 1147519: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1147520 - 1151615: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1151616 - 1155711: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1155712 - 1159807: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1159808 - 1163903: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1163904 - 1167999: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1168000 - 1168063: ................................................................ 1168064 - 1168127: ................................................................ 1168128 - 1168191: ................................................................ 1168192 - 1168255: ................................................................ 1168256 - 1168319: ................................................................ 1168320 - 1168383: ................................................................ 1168384 - 1168447: ................................................................ 1168448 - 1168511: ................................................................ 1168512 - 1168575: ................................................................ 1168576 - 1168639: ................................................................ 1168640 - 1168703: ................................................................ 1168704 - 1168767: ................................................................ 1168768 - 1168831: ................................................................ 1168832 - 1168895: ................................................................ 1168896 - 1168959: ................................................................ 1168960 - 1169023: ................................................................ 1169024 - 1169087: ................................................................ 1169088 - 1169151: ................................................................ 1169152 - 1169215: ................................................................ 1169216 - 1169279: ................................................................ 1169280 - 1169343: ................................................................ 1169344 - 1169407: ................................................................ 1169408 - 1169471: ................................................................ 1169472 - 1169535: ................................................................ 1169536 - 1169599: ................................................................ 1169600 - 1169663: ................................................................ 1169664 - 1169727: ................................................................ 1169728 - 1169791: ................................................................ 1169792 - 1169855: ................................................................ 1169856 - 1169919: ................................................................ 1169920 - 1169983: ................................................................ 1169984 - 1170047: ................................................................ 1170048 - 1170111: ................................................................ 1170112 - 1170175: ................................................................ 1170176 - 1170239: ................................................................ 1170240 - 1170303: ................................................................ 1170304 - 1170367: ................................................................ 1170368 - 1170431: ................................................................ 1170432 - 1170495: ................................................................ 1170496 - 1170559: ................................................................ 1170560 - 1170623: ................................................................ 1170624 - 1170687: ................................................................ 1170688 - 1170751: ................................................................ 1170752 - 1170815: ................................................................ 1170816 - 1170879: ................................................................ 1170880 - 1170943: ................................................................ 1170944 - 1171007: ................................................................ 1171008 - 1171071: ................................................................ 1171072 - 1171135: ................................................................ 1171136 - 1171199: ................................................................ 1171200 - 1171263: ................................................................ 1171264 - 1171327: ................................................................ 1171328 - 1171391: ................................................................ 1171392 - 1171455: ................................................................ 1171456 - 1171519: ................bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1171520 - 1171583: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1171584 - 1171647: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1171648 - 1171711: bb......bb......bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1171712 - 1171775: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.......bbb.....bbb.....bb...... 1171776 - 1171839: bbbbbbbbbbbbbbbbbbbbbbbbbb...bbbbb......bbbbbbbbbbbbbb.bbbbbbbbb 1171840 - 1171903: bbbbb..bbb....b.bb.bbbbbbbbbbbbbbbbbb...bb......bbbbbb..bbbbbbbb 1171904 - 1171967: bbbbbbbbb......bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1171968 - 1172031: bbbbbbbbbbbbbbbb................................................ 1172032 - 1172095: ................................................................ 1172096 - 1176191: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1176192 - 1180287: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1180288 - 1184383: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1184384 - 1188479: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1188480 - 1192575: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1192576 - 1196671: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1196672 - 1200767: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1200768 - 1204863: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1204864 - 1208959: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1208960 - 1213055: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1213056 - 1217151: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1217152 - 1221247: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1221248 - 1225343: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1225344 - 1229439: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1229440 - 1233535: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1233536 - 1237631: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1237632 - 1241727: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1241728 - 1245823: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1245824 - 1249919: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1249920 - 1254015: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1254016 - 1258111: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1258112 - 1262207: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1262208 - 1266303: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1266304 - 1270399: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1270400 - 1274495: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1274496 - 1278591: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1278592 - 1282687: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1282688 - 1286783: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1286784 - 1290879: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1290880 - 1294975: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1294976 - 1299071: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1299072 - 1303167: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1303168 - 1307263: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1307264 - 1311359: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1311360 - 1315455: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1315456 - 1319551: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1319552 - 1323647: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1323648 - 1323711: ................................................................ 1323712 - 1323775: ................................................................ 1323776 - 1323839: ................................................................ 1323840 - 1323903: ................................................................ 1323904 - 1323967: ................................................................ 1323968 - 1324031: ................................................................ 1324032 - 1324095: ................................................................ 1324096 - 1324159: ................................................................ 1324160 - 1324223: ................................................................ 1324224 - 1324287: ................................................................ 1324288 - 1324351: ................................................................ 1324352 - 1324415: ................................................................ 1324416 - 1324479: ................................................................ 1324480 - 1324543: ................................................................ 1324544 - 1324607: ................................................................ 1324608 - 1324671: ................................................................ 1324672 - 1324735: ................................................................ 1324736 - 1324799: ................................................................ 1324800 - 1324863: ................................................................ 1324864 - 1324927: ................................................................ 1324928 - 1324991: ................................................................ 1324992 - 1325055: ................................................................ 1325056 - 1325119: ................................................................ 1325120 - 1325183: ................................................................ 1325184 - 1325247: ................................................................ 1325248 - 1325311: ................................................................ 1325312 - 1325375: ................................................................ 1325376 - 1325439: ................................................................ 1325440 - 1325503: ................................................................ 1325504 - 1325567: ................................................................ 1325568 - 1325631: ................................................................ 1325632 - 1325695: ................................................................ 1325696 - 1325759: ................................................................ 1325760 - 1325823: ................................................................ 1325824 - 1325887: ................................................................ 1325888 - 1325951: ................................................................ 1325952 - 1326015: ................................................................ 1326016 - 1326079: ................................................................ 1326080 - 1326143: ................................................................ 1326144 - 1326207: ................................................................ 1326208 - 1326271: ................................................................ 1326272 - 1326335: ................................................................ 1326336 - 1326399: ................................................................ 1326400 - 1326463: ................................................................ 1326464 - 1326527: ................................................................ 1326528 - 1326591: ................................................................ 1326592 - 1326655: ................................................................ 1326656 - 1326719: ................................................................ 1326720 - 1326783: ................................................................ 1326784 - 1326847: ................................................................ 1326848 - 1326911: ................................................................ 1326912 - 1326975: ................................................................ 1326976 - 1327039: ................................................................ 1327040 - 1327103: ................................................................ 1327104 - 1327167: ................bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327168 - 1327231: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327232 - 1327295: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327296 - 1327359: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327360 - 1327423: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327424 - 1327487: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.....bbbbbbbbbbbbbbbbbbbbbbbbb 1327488 - 1327551: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.....bbb.....bbbb....bbb..... 1327552 - 1327615: bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb 1327616 - 1327679: bbbbbbbbbbbbbbbb................................................ 1327680 - 1327743: ................................................................ 1327744 - 1589887: OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO 1589888 - 1593983: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1593984 - 1598079: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1598080 - 1602175: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1602176 - 1606271: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1606272 - 1610367: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1610368 - 1614463: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1614464 - 1618559: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1618560 - 1622655: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1622656 - 1626751: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1626752 - 1630847: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1630848 - 1634943: oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo 1634944 - 1635007: ................................................................ 1635008 - 1635071: ................................................................ 1635072 - 1635135: ................................................................ 1635136 - 1635199: ................................................................ 1635200 - 1635263: ................................................................ 1635264 - 1635327: ................................................................ 1635328 - 1635391: ................................................................ 1635392 - 1635455: ................................................................ 1635456 - 1635519: ................................................................ 1635520 - 1635583: ................................................................ 1635584 - 1635647: ................................................................ 1635648 - 1635711: ................................................................ 1635712 - 1635775: ................................................................ 1635776 - 1635839: ................................................................ 1635840 - 1635903: ................................................................ 1635904 - 1635967: ................................................................ 1635968 - 1636031: ................................................................ 1636032 - 1636095: ................................................................ 1636096 - 1636159: ................................................................ 1636160 - 1636223: ................................................................ 1636224 - 1636287: ................................................................ 1636288 - 1636351: ................................................................ 1636352 - 1636415: ................................................................ 1636416 - 1636479: ................................................................ 1636480 - 1636543: ................................................................ 1636544 - 1636607: ................................................................ 1636608 - 1636671: ................................................................ 1636672 - 1636735: ................................................................ 1636736 - 1636799: ................................................................ 1636800 - 1636863: ................................................................ 1636864 - 1636927: ................................................................ 1636928 - 1636991: ................................................................ 1636992 - 1637055: ................................................................ 1637056 - 1637119: ................................................................ 1637120 - 1637183: ................................................................ 1637184 - 1637247: ................................................................ 1637248 - 1637311: ................................................................ 1637312 - 1637375: ................................................................ 1637376 - 1637439: ................................................................ 1637440 - 1637503: ................................................................ 1637504 - 1637567: ................................................................ 1637568 - 1637631: ................................................................ 1637632 - 1637695: ................................................................ 1637696 - 1637759: ................................................................ 1637760 - 1637823: ................................................................ 1637824 - 1637887: ................................................................ 1637888 - 1637951: ................................................................ 1637952 - 1638015: ................................................................ 1638016 - 1638079: ................................................................ 1638080 - 1638143: ................................................................ 1638144 - 1638207: ................................................................ 1638208 - 1638271: ................................................................ 1638272 - 1638335: ................................................................ 1638336 - 1638399: ................................................................ Resuming helter_skelter/syncd_primary_3legs_3 1254 blocks differ 0.0765380859375% of the mirror is bad.
Reproduced this again last night with only two mirrors (but again both with three legs). This test now verifies that all legs are in sync (instead of just the last one created) before doing any leg failure operations. Again, it's the last mirror which becomes corrupt. [root@link-02 ~]# lvs -a -o +devices /dev/sdg1: open failed: No such device or address LV VG Attr LSize Origin Snap% Move Log Copy% Devices LogVol00 VolGroup00 -wi-ao 35.16G /dev/hda2(0) LogVol01 VolGroup00 -wi-ao 1.94G /dev/hda2(1125) syncd_primary_3legs_1 helter_skelter mwi-ao 800.00M syncd_primary_3legs_1_mlog 100.00 syncd_primary_3legs_1_mimage_2(0),syncd_primary_3legs_1_mimage_1(0) [syncd_primary_3legs_1_mimage_1] helter_skelter iwi-ao 800.00M /dev/sdc1(0) [syncd_primary_3legs_1_mimage_2] helter_skelter iwi-ao 800.00M /dev/sdd1(0) [syncd_primary_3legs_1_mlog] helter_skelter lwi-ao 4.00M /dev/sde1(0) syncd_primary_3legs_2 helter_skelter mwi-ao 800.00M syncd_primary_3legs_2_mlog 100.00 syncd_primary_3legs_2_mimage_2(0),syncd_primary_3legs_2_mimage_1(0) [syncd_primary_3legs_2_mimage_1] helter_skelter iwi-ao 800.00M /dev/sdc1(200) [syncd_primary_3legs_2_mimage_2] helter_skelter iwi-ao 800.00M /dev/sdd1(200) [syncd_primary_3legs_2_mlog] helter_skelter lwi-ao 4.00M /dev/sde1(1) [root@link-02 ~]# touch /mnt/syncd_primary_3legs_1/foo [root@link-02 ~]# touch /mnt/syncd_primary_3legs_2/foo touch: cannot touch `/mnt/syncd_primary_3legs_2/foo': Input/output error I'll run 'find_dm_diffs' on both the good mirror and the bad mirror and attach the results.
Created attachment 245571 [details] this is the non corrupt mirror (syncd_primary_3legs_1)
Created attachment 245581 [details] this is the corrupt mirror (syncd_primary_3legs_2)
Jon, Should this be a candidate for a 4.6.Z errata like 290821?
Any bug that has been shown to corrupt data certainly gets first priority. Let's try to limit the scope. 1) Can you make this happen with just one mirror? 2) Can you make this happen with two-way mirrors? (You won't be able to use the checker script, but GFS will obviously still complain.) 3) Are the servers for the mirrors located on the same or different machines? (dmsetup table ; <view> /var/log/messages) Is there a pattern that can be linked to the last mirror, as that seems to be the one with the problem?
> 1) Can you make this happen with just one mirror? No, it appears that you need at least 2 > 2) Can you make this happen with two-way mirrors? No, it appears that you need 3 legs > 3) Are the servers for the mirrors located on the same or different machines? I've seen it where the servers were on different nodes and where they were on the same node
Created attachment 257231 [details] updated test allows specification of devices. Able to reproduce with this new script using the following args: $> diff_test.sh -m 2 -f 3 -S -d /dev/sda,/dev/sdb,/dev/sdc,/dev/sdd:0-50 Once I/O starts, I kill the primary
reproducible with single machine mirroring
*** Bug 243013 has been marked as a duplicate of this bug. ***
Created attachment 300084 [details] Updated 'diff_test.sh' Updated test
3-way mirrors are not supported on RHEL4. Closing WONTFIX. Please try to reproduce on later releases where 3-way mirrors are supported.