Bug 809231 - merged back raid image (with change tracking) doesn't appear to get synced properly
merged back raid image (with change tracking) doesn't appear to get synced pr...
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.3
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Jonathan Earl Brassow
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-02 15:16 EDT by Corey Marthaler
Modified: 2012-06-20 04:45 EDT (History)
10 users (show)

See Also:
Fixed In Version: kernel-2.6.32-269.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-20 04:45:15 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2012:0862 normal SHIPPED_LIVE Moderate: Red Hat Enterprise Linux 6 kernel security, bug fix and enhancement update 2012-06-20 08:55:00 EDT

  None (edit)
Description Corey Marthaler 2012-04-02 15:16:28 EDT
Description of problem:

SCENARIO - [split_w_tracking_io_merge]
Create a 3-way raid1 with fs data, verify data, split image with tracking, change data on raid vol, merge split image data back, then verify again
taft-01: lvcreate --type raid1 -m 2 -n split_tracking -L 1G split_image
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 27.23% )
   0/1 mirror(s) are fully synced: ( 61.99% )
   1/1 mirror(s) are fully synced: ( 100.00% )

Placing an ext filesystem on raid1 volume
mke2fs 1.41.12 (17-May-2010)
Mounting raid1 volume


# INITIAL WRITE AND VERIFY
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.27391 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        25299
Verify XIOR Stream: /tmp/split_trackingA.27391
Working dir:        /mnt/split_tracking

Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.27391
Working dir:        /mnt/split_tracking


# ISSUE A SYNC DUE TO BUG 808189
Issuing a sync to force data to disk
splitting off leg from raid with tracking...
taft-01: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking


# VERIFY INITIAL DATA ON SPLIT LEG
+++ Mounting and verifying split image data +++
mount: block device /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.27391
Working dir:        /mnt/split_tracking_rimage_2


# ADD NEW DATA TO MIRROR W/ SPLIT OFF LEG
Writing new data to the raid and then merging back the split off image
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.27391 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        25360
Verify XIOR Stream: /tmp/split_trackingB.27391
Working dir:        /mnt/split_tracking
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingB.27391
Working dir:        /mnt/split_tracking
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.27391 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        25364
Verify XIOR Stream: /tmp/split_trackingC.27391
Working dir:        /mnt/split_tracking
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingC.27391
Working dir:        /mnt/split_tracking



# ADD SPLIT OFF LEG BACK TO RAID (ASSUMPTION IS THAT IT WOULD BE PUT BACK IN SYNC WITH OTHER LEGS)
Merge split off image split_image/split_tracking_rimage_2 back into the raid
lvconvert --merge split_image/split_tracking_rimage_2
out=  split_image/split_tracking_rimage_2 successfully merged back into split_image/split_tracking


# SPLIT OFF SAME IMAGE AGAIN AND VERIFY OLD AND NEW DATA
Issuing a sync to force data to disk
AGAIN, splitting off leg from raid with tracking...
taft-01: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

# INITIAL DATA IS THERE
+++ Mounting and verifying split image data +++
mount: block device /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.27391
Working dir:        /mnt/split_tracking_rimage_2


# THE NEW DATA IS NOT HOWEVER
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.27391 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingB.27391
Working dir:        /mnt/split_tracking_rimage_2
Can not stat rbjooktiubqtktnuskxsympjhuuyggjxhu: No such file or directory
checkit verify failed

Apr  2 14:06:37 taft-01 qarshd[25477]: Running cmdline: /usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trv
Apr  2 14:06:37 taft-01 kernel: attempt to access beyond end of device
Apr  2 14:06:37 taft-01 kernel: dm-7: rw=0, want=7545668040, limit=2097152


Version-Release number of selected component (if applicable):
2.6.32-251.el6.x86_64

lvm2-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
lvm2-libs-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
lvm2-cluster-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-libs-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-event-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
device-mapper-event-libs-1.02.74-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012
cmirror-2.02.95-2.el6    BUILT: Fri Mar 16 08:39:54 CDT 2012


How reproducible:
Everytime
Comment 2 RHEL Product and Program Management 2012-04-18 15:30:09 EDT
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.
Comment 3 Jonathan Earl Brassow 2012-04-18 18:11:25 EDT
Before patch:
[root@bp-01 ~]# lvcreate --type raid1 -m 2 -L 1G -n lv vg
  Logical volume "lv" created
[root@bp-01 ~]# devices vg
  LV            Copy%  Devices                                     
  lv            100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]        /dev/sde1(1)                                
  [lv_rimage_1]        /dev/sdc1(1)                                
  [lv_rimage_2]        /dev/sdf1(1)                                
  [lv_rmeta_0]         /dev/sde1(0)                                
  [lv_rmeta_1]         /dev/sdc1(0)                                
  [lv_rmeta_2]         /dev/sdf1(0)                                
[root@bp-01 ~]# lvconvert --splitmirrors 1 --trackchanges vg/lv
  lv_rimage_2 split from lv for read-only purposes.
  Use 'lvconvert --merge vg/lv_rimage_2' to merge back into lv
[root@bp-01 ~]# dd if=/dev/zero of=/dev/vg/lv bs=4M
dd: writing `/dev/vg/lv': No space left on device
257+0 records in
256+0 records out
1073741824 bytes (1.1 GB) copied, 241.313 s, 4.4 MB/s
[root@bp-01 ~]# sync
[root@bp-01 ~]# lvconvert --merge vg/lv_rimage_2; dmsetup status vg-lv
  vg/lv_rimage_2 successfully merged back into vg/lv
0 2097152 raid raid1 3 AAA 2097152/2097152
^^^^^^^^^^^^^^^^ SHOULD NOT BE FULLY SYNC'ED - INDICATES RECOVERY IS SKIPPED


After patch:
[root@bp-01 ~]# lvcreate --type raid1 -m 2 -L 1G -n lv vg
  Logical volume "lv" created
[root@bp-01 ~]# devices vg
  LV            Copy%  Devices                                     
  lv            100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
  [lv_rimage_0]        /dev/sde1(1)                                
  [lv_rimage_1]        /dev/sdc1(1)                                
  [lv_rimage_2]        /dev/sdf1(1)                                
  [lv_rmeta_0]         /dev/sde1(0)                                
  [lv_rmeta_1]         /dev/sdc1(0)                                
  [lv_rmeta_2]         /dev/sdf1(0)                                
[root@bp-01 ~]# lvconvert --splitmirrors 1 --trackchanges vg/lv
  lv_rimage_2 split from lv for read-only purposes.
  Use 'lvconvert --merge vg/lv_rimage_2' to merge back into lv
[root@bp-01 ~]# dd if=/dev/zero of=/dev/vg/lv bs=4M
dd: writing `/dev/vg/lv': No space left on device
257+0 records in
256+0 records out
1073741824 bytes (1.1 GB) copied, 237.336 s, 4.5 MB/s
[root@bp-01 ~]# sync
[root@bp-01 ~]# lvconvert --merge vg/lv_rimage_2; dmsetup status vg-lv
  vg/lv_rimage_2 successfully merged back into vg/lv
0 2097152 raid raid1 3 AAa 0/2097152
^^^^^^^^^^^^^^^^^^ Device is properly signified as out-of-sync

[root@bp-01 ~]# dmsetup status vg-lv
0 2097152 raid raid1 3 AAA 2097152/2097152
^^^^^^^^^^^^^^^^^^ And goes on to recover properly.
Comment 4 Jarod Wilson 2012-05-02 12:20:04 EDT
Patch(es) available on kernel-2.6.32-269.el6
Comment 7 Corey Marthaler 2012-05-02 19:41:17 EDT
Fix appears to work in the latest kernel.


2.6.32-269.el6.x86_64
lvm2-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
lvm2-libs-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
lvm2-cluster-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
udev-147-2.41.el6    BUILT: Thu Mar  1 13:01:08 CST 2012
device-mapper-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-libs-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-event-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
device-mapper-event-libs-1.02.74-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012
cmirror-2.02.95-7.el6    BUILT: Wed May  2 05:14:03 CDT 2012


SCENARIO - [split_w_tracking_io_merge]
Create a 3-way raid1 with fs data, verify data, split image with tracking, change data on raid vol, merge split image data back, verify origin data
taft-01: lvcreate --type raid1 -m 2 -n split_tracking -L 1G split_image
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 28.72% )
   0/1 mirror(s) are fully synced: ( 61.51% )
   0/1 mirror(s) are fully synced: ( 94.91% )
   1/1 mirror(s) are fully synced: ( 100.00% )

Placing an ext filesystem on raid1 volume
mke2fs 1.41.12 (17-May-2010)
Mounting raid1 volume

Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.26258 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        17282
Verify XIOR Stream: /tmp/split_trackingA.26258
Working dir:        /mnt/split_tracking

Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.26258
Working dir:        /mnt/split_tracking


Issuing a sync to force data to disk
splitting off leg from raid with tracking...
taft-01: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

+++ Mounting and verifying split image data +++
mount: block device /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.26258
Working dir:        /mnt/split_tracking_rimage_2

Writing new data to the raid and then merging back the split off image
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.26258 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        17347
Verify XIOR Stream: /tmp/split_trackingB.26258
Working dir:        /mnt/split_tracking
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingB.26258
Working dir:        /mnt/split_tracking
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.26258 -n 500
checkit starting with:
CREATE
Num files:          500
Random Seed:        17351
Verify XIOR Stream: /tmp/split_trackingC.26258
Working dir:        /mnt/split_tracking
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingC.26258
Working dir:        /mnt/split_tracking
Merge split off image split_image/split_tracking_rimage_2 back into the raid
lvconvert --merge split_image/split_tracking_rimage_2

Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Issuing a sync to force data to disk
AGAIN, splitting off leg from raid with tracking...
taft-01: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

+++ Mounting and verifying split image data +++
mount: block device /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingA.26258
Working dir:        /mnt/split_tracking_rimage_2
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingB.26258
Working dir:        /mnt/split_tracking_rimage_2
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel6.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingC.26258 -v
checkit starting with:
VERIFY
Verify XIOR Stream: /tmp/split_trackingC.26258
Working dir:        /mnt/split_tracking_rimage_2

Merge split off image split_image/split_tracking_rimage_2 back so it can be deleted
lvconvert --merge split_image/split_tracking_rimage_2

Deactivating mirror split_tracking... and removing
Comment 9 errata-xmlrpc 2012-06-20 04:45:15 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html

Note You need to log in before you can comment on or make changes to this bug.