Bug 1090101

Summary: RAID LV leg not fully synced after split and merge
Product: Red Hat Enterprise Linux 7 Reporter: Marian Csontos <mcsontos>
Component: lvm2Assignee: Heinz Mauelshagen <heinzm>
lvm2 sub component: Mirroring and RAID QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: unspecified    
Priority: unspecified CC: agk, cmarthal, heinzm, jbrassow, lmiksik, msnitzer, prajnoha, prockai, zkabelac
Version: 7.0   
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: lvm2-2.02.165-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 04:07:43 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Marian Csontos 2014-04-22 15:12:48 UTC
Description of problem:
splitting and merging a LV-RAID leg sometimes fails to update the leg properly.
Nothing in logs indicating failure, just the data are not there.

Version-Release number of selected component (if applicable):
lvm2-2.02.105-14.el7.x86_64
kernel-3.10.0-121.el7.x86_64
(lvmetad is running, so far 0 reproducibility without lvmetad)

How reproducible:
low

Steps to Reproduce:

    pvcreate /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 /dev/sdg1 /dev/sdj1
    vgcreate split_image /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 /dev/sdg1 /dev/sdj1
    pvs --noheadings
    lvcreate --type raid1 -m 2 -n split_tracking -L 1G split_image
# WAIT FOR SYNC...
    dmsetup info
    mkfs /dev/split_image/split_tracking
    mkdir -p /mnt/split_tracking
    mount /dev/split_image/split_tracking /mnt/split_tracking
# WRITE1 AND VERIFY:
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -n 500
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -v
    sync
    lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking
    mkdir -p /mnt/split_tracking_rimage_2
    mount /dev/split_image/split_tracking_rimage_2 /mnt/split_tracking_rimage_2
# VERIFY SPLIT LEG:
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.31800 -v
    umount /mnt/split_tracking_rimage_2
# WRITE TO ORIGIN AND VERIFY:
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.31800 -n 500
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.31800 -v
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.31800 -n 500
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.31800 -v
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -v
    sync
    lvconvert --merge split_image/split_tracking_rimage_2
# WAIT FOR SYNC...
    dmsetup info
    sync
    lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking
    mkdir -p /mnt/split_tracking_rimage_2
    mount /dev/split_image/split_tracking_rimage_2 /mnt/split_tracking_rimage_2
# VERIFY THE LEG WAS SYNCED:
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.31800 -v
    /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.31800 -v

Actual results:
The RAID LV contains correct data, but the leg was not synced.

Expected results:
All legs should be in sync.

Additional info:

Comment 2 RHEL Program Management 2014-04-30 05:47:35 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.

Comment 5 Heinz Mauelshagen 2016-09-05 20:19:41 UTC
Tested ok with kernel 3.10.0-501.el7.x86_64 and
  LVM version:     2.02.164(2)-RHEL7 (2016-08-31)
  Library version: 1.02.133-RHEL7 (2016-08-31)
  Driver version:  4.34.0

Comment 7 Corey Marthaler 2016-09-09 22:17:38 UTC
Marking verified (SanityOnly). I was never able to reproduce this issue before the fix.


3.10.0-501.el7.x86_64
lvm2-2.02.165-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
lvm2-libs-2.02.165-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
lvm2-cluster-2.02.165-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
device-mapper-1.02.134-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
device-mapper-libs-1.02.134-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
device-mapper-event-1.02.134-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
device-mapper-event-libs-1.02.134-1.el7    BUILT: Wed Sep  7 11:04:22 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016



[root@host-118 ~]# systemctl status lvm2-lvmetad
 lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: enabled)
   Active: active (running) since Wed 2016-09-07 19:01:17 CDT; 1 day 22h ago
     Docs: man:lvmetad(8)
 Main PID: 472 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           /usr/sbin/lvmetad -f



creating lvm devices...
host-118: pvcreate /dev/sdf1 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdg1
host-118: vgcreate  split_image /dev/sdf1 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdg1

============================================================
Iteration 1 of 1 started at Fri Sep  9 16:45:25 CDT 2016
============================================================
SCENARIO - [split_w_tracking_io_merge]
Create a 3-way raid1 with fs data, verify data, split image with tracking, change data on raid vol, merge split image data back, verify origin data
host-118: lvcreate  --type raid1 -m 2 -n split_tracking -L 1G split_image
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Placing an ext filesystem on raid1 volume
mke2fs 1.42.9 (28-Dec-2013)
Mounting raid1 volume

Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -n 500
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -v

Issuing a sync to force data to disk
splitting off leg from raid with tracking...
host-118: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

  LV                        VG          Attr       LSize Cpy%Sync Devices
  split_tracking            split_image rwi-aor--- 1.00g 100.00   split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
  [split_tracking_rimage_0] split_image iwi-aor--- 1.00g          /dev/sdf1(1)
  [split_tracking_rimage_1] split_image iwi-aor--- 1.00g          /dev/sda1(1)
  split_tracking_rimage_2   split_image Iri-a-r--- 1.00g          /dev/sdb1(1)
  [split_tracking_rmeta_0]  split_image ewi-aor--- 4.00m          /dev/sdf1(0)
  [split_tracking_rmeta_1]  split_image ewi-aor--- 4.00m          /dev/sda1(0)
  [split_tracking_rmeta_2]  split_image ewi-a-r--- 4.00m          /dev/sdb1(0)

 split_image-split_tracking: 0 2097152 raid raid1 3 AA 2097152/2097152 idle 0 0

+++ Mounting and verifying split image data +++
mount: /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.3276 -v

Writing new data to the raid and then merging back the split off image
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.3276 -n 500
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.3276 -n 500
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -v
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.3276 -v
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.3276 -v


Issuing a sync to force data to disk
Merge split off image split_image/split_tracking_rimage_2 back into the raid
lvconvert --merge split_image/split_tracking_rimage_2
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
  LV                        VG          Attr       LSize Cpy%Sync Devices
  split_tracking            split_image rwi-aor--- 1.00g 100.00   split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
  [split_tracking_rimage_0] split_image iwi-aor--- 1.00g          /dev/sdf1(1)
  [split_tracking_rimage_1] split_image iwi-aor--- 1.00g          /dev/sda1(1)
  [split_tracking_rimage_2] split_image iwi-aor--- 1.00g          /dev/sdb1(1)
  [split_tracking_rmeta_0]  split_image ewi-aor--- 4.00m          /dev/sdf1(0)
  [split_tracking_rmeta_1]  split_image ewi-aor--- 4.00m          /dev/sda1(0)
  [split_tracking_rmeta_2]  split_image ewi-aor--- 4.00m          /dev/sdb1(0)

 split_image-split_tracking: 0 2097152 raid raid1 3 AAA 2097152/2097152 idle 0 0

Issuing a sync to force data to disk
AGAIN, splitting off leg from raid with tracking...
host-118: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking

Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

  LV                        VG          Attr       LSize Cpy%Sync Devices
  split_tracking            split_image rwi-aor--- 1.00g 100.00   split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
  [split_tracking_rimage_0] split_image iwi-aor--- 1.00g          /dev/sdf1(1)
  [split_tracking_rimage_1] split_image iwi-aor--- 1.00g          /dev/sda1(1)
  split_tracking_rimage_2   split_image Iri-a-r--- 1.00g          /dev/sdb1(1)
  [split_tracking_rmeta_0]  split_image ewi-aor--- 4.00m          /dev/sdf1(0)
  [split_tracking_rmeta_1]  split_image ewi-aor--- 4.00m          /dev/sda1(0)
  [split_tracking_rmeta_2]  split_image ewi-a-r--- 4.00m          /dev/sdb1(0)

 split_image-split_tracking: 0 2097152 raid raid1 3 AA 2097152/2097152 idle 0 0

+++ Mounting and verifying split image data +++
mount: /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.3276 -v
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.3276 -v
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingC.3276 -v

Merge split off image split_image/split_tracking_rimage_2 back so it can be deleted
lvconvert --merge split_image/split_tracking_rimage_2
  LV                        VG          Attr       LSize Cpy%Sync Devices
  split_tracking            split_image rwi-a-r--- 1.00g 100.00   split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
  [split_tracking_rimage_0] split_image iwi-aor--- 1.00g          /dev/sdf1(1)
  [split_tracking_rimage_1] split_image iwi-aor--- 1.00g          /dev/sda1(1)
  [split_tracking_rimage_2] split_image iwi-aor--- 1.00g          /dev/sdb1(1)
  [split_tracking_rmeta_0]  split_image ewi-aor--- 4.00m          /dev/sdf1(0)
  [split_tracking_rmeta_1]  split_image ewi-aor--- 4.00m          /dev/sda1(0)
  [split_tracking_rmeta_2]  split_image ewi-aor--- 4.00m          /dev/sdb1(0)

 split_image-split_tracking: 0 2097152 raid raid1 3 AAA 2097152/2097152 idle 0 0

Deactivating LV split_image/split_tracking on host-118... and removing

Comment 9 errata-xmlrpc 2016-11-04 04:07:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-1445.html