Bug 998730

Summary: Should split off raid images (with tracking) have out-of-sync lv_attrs?
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.5CC: agk, dwysocha, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, prockai, thornber, tlavigne, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.100-6.el6 Doc Type: Bug Fix
Doc Text:
no doc text needed.
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 23:27:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2013-08-19 22:49:32 UTC
Description of problem:
Once I split off a raid image, it's lv_attr doesn't change to "out-of-sync" until it has been deactivated. Even a write to the initial raid volume will not cause it to change.

# From lvs man page: 
mirror or raid (i)mage, mirror or raid (I)mage out-of-sync

SCENARIO - [split_w_tracking_sanity]
Create a 3-way raid1 and split it with tracking and verify sanity
harding-03: lvcreate --type raid1 -m 2 -n split_tracking -L 500M split_image
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 87.96% )
   1/1 mirror(s) are fully synced: ( 100.00% )

splitting off leg from raid with tracking...
lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

 LV                        Attr       LSize   Cpy%Sync Devices
 split_tracking            rwi-a-r--- 500.00m   100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
 [split_tracking_rimage_0] iwi-aor--- 500.00m          /dev/sdb3(1)
 [split_tracking_rimage_1] iwi-aor--- 500.00m          /dev/sdb2(1)
 split_tracking_rimage_2   iri-a-r--- 500.00m          /dev/sdb1(1) <- "insync"
 [split_tracking_rmeta_0]  ewi-aor---   4.00m          /dev/sdb3(0)
 [split_tracking_rmeta_1]  ewi-aor---   4.00m          /dev/sdb2(0)
 [split_tracking_rmeta_2]  ewi-a-r---   4.00m          /dev/sdb1(0)


[root@harding-03 lvm]# dd if=/dev/zero of=/dev/split_image/split_tracking count=5
5+0 records in
5+0 records out
2560 bytes (2.6 kB) copied, 0.00531551 s, 482 kB/s
[root@harding-03 lvm]# sync
[root@harding-03 lvm]# lvs -a -o +devices
 LV                        Attr       LSize   Cpy%Sync Devices
 split_tracking            rwi-a-r--- 500.00m   100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
 [split_tracking_rimage_0] iwi-aor--- 500.00m          /dev/sdb3(1)
 [split_tracking_rimage_1] iwi-aor--- 500.00m          /dev/sdb2(1)
 split_tracking_rimage_2   iri-a-r--- 500.00m          /dev/sdb1(1) <- "insync"
 [split_tracking_rmeta_0]  ewi-aor---   4.00m          /dev/sdb3(0)
 [split_tracking_rmeta_1]  ewi-aor---   4.00m          /dev/sdb2(0)
 [split_tracking_rmeta_2]  ewi-a-r---   4.00m          /dev/sdb1(0)

[root@harding-03 lvm]# lvchange -an split_image/split_tracking_rimage_2
[root@harding-03 lvm]# lvs -a -o +devices
 LV                        Attr       LSize   Cpy%Sync Devices
 split_tracking            rwi-a-r--- 500.00m   100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
 [split_tracking_rimage_0] iwi-aor--- 500.00m          /dev/sdb3(1)
 [split_tracking_rimage_1] iwi-aor--- 500.00m          /dev/sdb2(1)
 split_tracking_rimage_2   Iri---r--- 500.00m          /dev/sdb1(1) <- "out-of-sync"
 [split_tracking_rmeta_0]  ewi-aor---   4.00m          /dev/sdb3(0)
 [split_tracking_rmeta_1]  ewi-aor---   4.00m          /dev/sdb2(0)
 [split_tracking_rmeta_2]  ewi-a-r---   4.00m          /dev/sdb1(0)


Version-Release number of selected component (if applicable):
2.6.32-410.el6.x86_64
lvm2-2.02.100-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
lvm2-libs-2.02.100-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
lvm2-cluster-2.02.100-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
udev-147-2.48.el6    BUILT: Fri Aug  9 06:09:50 CDT 2013
device-mapper-1.02.79-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
device-mapper-libs-1.02.79-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
device-mapper-event-1.02.79-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
device-mapper-event-libs-1.02.79-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013
cmirror-2.02.100-2.el6    BUILT: Wed Aug 14 10:23:33 CDT 2013

Comment 2 Marian Csontos 2013-09-05 09:30:42 UTC
Every now and then I see bad data on a leg which was split, raid was written to, the leg was merged and split again.

Jon, is it possible this would cause such problems when merging the leg back?

If so, I believe this is high/urgent severity bug, as it may lead to data corruption and must be solved for 6.5.

Otherwise, that's another issue and I will open a Blocker for 6.5.
But as this one looks rather easy to solve, could we start checking this?

-- Marian

Comment 3 RHEL Program Management 2013-10-14 02:39:55 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 5 Jonathan Earl Brassow 2013-10-14 14:53:43 UTC
(In reply to Marian Csontos from comment #2)
> Every now and then I see bad data on a leg which was split, raid was written
> to, the leg was merged and split again.
> 
> Jon, is it possible this would cause such problems when merging the leg back?
> 
> If so, I believe this is high/urgent severity bug, as it may lead to data
> corruption and must be solved for 6.5.
> 
> Otherwise, that's another issue and I will open a Blocker for 6.5.
> But as this one looks rather easy to solve, could we start checking this?
> 
> -- Marian

That would be a bug.  However, please remember that the split does not work like snapshot.  It does not quiesce the file system before performing the action.  In other words, you must at a minimum perform a "sync" before splitting and better yet perform an unmount of the file system before splitting.  This is part of the documentation WRT the splitmirror operation.

Comment 6 Jonathan Earl Brassow 2013-10-14 15:38:36 UTC
Yes, the split image should have an out-of-sync attr ('I') - always.

Even it the RAID LV has not been written to since the LV was split off, it is still not part of the group that makes up the RAID and is therefore "out-of-sync".

The attr should therefore always be 'I'.

Comment 7 Jonathan Earl Brassow 2013-10-14 15:51:27 UTC
Fix committed upstream as:

commit f58b26b6338b2939e6a88174ac2dc185b345118b
Author: Jonathan Brassow <jbrassow>
Date:   Mon Oct 14 10:48:44 2013 -0500

    RAID: Report RAID images split with tracking as out-of-sync ("I").
    
    Split image should have an out-of-sync attr ('I') - always.  Even if
    the RAID LV has not been written to since the LV was split off, it is
    still not part of the group that makes up the RAID and is therefore
    "out-of-sync".

Comment 11 Corey Marthaler 2013-10-18 18:36:15 UTC
Fix verified in the latest rpms.

SCENARIO - [split_w_tracking_sanity]
Create a 3-way raid1 and split it with tracking and verify sanity
harding-02: lvcreate --type raid1 -m 2 -n split_tracking -L 500M split_image
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 93.50% )
   1/1 mirror(s) are fully synced: ( 100.00% )

splitting off leg from raid with tracking...
 LV                        Attr       LSize    Cpy%Sync Devices
 split_tracking            rwi-a-r--- 500.00m    100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
 [split_tracking_rimage_0] iwi-aor--- 500.00m           /dev/sdb3(1)
 [split_tracking_rimage_1] iwi-aor--- 500.00m           /dev/sdb2(1)
 split_tracking_rimage_2   Iri-a-r--- 500.00m           /dev/sdb1(1)  <-------
 [split_tracking_rmeta_0]  ewi-aor---   4.00m           /dev/sdb3(0)
 [split_tracking_rmeta_1]  ewi-aor---   4.00m           /dev/sdb2(0)
 [split_tracking_rmeta_2]  ewi-a-r---   4.00m           /dev/sdb1(0)


2.6.32-410.el6.x86_64
lvm2-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
lvm2-libs-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
lvm2-cluster-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
udev-147-2.50.el6    BUILT: Fri Oct 11 05:58:10 CDT 2013
device-mapper-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-libs-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-event-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
device-mapper-event-libs-1.02.79-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013
cmirror-2.02.100-6.el6    BUILT: Wed Oct 16 07:26:00 CDT 2013

Comment 12 errata-xmlrpc 2013-11-21 23:27:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1704.html