Bug 1090101
| Summary: | RAID LV leg not fully synced after split and merge | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Marian Csontos <mcsontos> |
| Component: | lvm2 | Assignee: | Heinz Mauelshagen <heinzm> |
| lvm2 sub component: | Mirroring and RAID | QA Contact: | cluster-qe <cluster-qe> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | agk, cmarthal, heinzm, jbrassow, lmiksik, msnitzer, prajnoha, prockai, zkabelac |
| Version: | 7.0 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | lvm2-2.02.165-1.el7 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-04 04:07:43 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This request was not resolved in time for the current release. Red Hat invites you to ask your support representative to propose this request, if still desired, for consideration in the next release of Red Hat Enterprise Linux. Tested ok with kernel 3.10.0-501.el7.x86_64 and LVM version: 2.02.164(2)-RHEL7 (2016-08-31) Library version: 1.02.133-RHEL7 (2016-08-31) Driver version: 4.34.0 Marking verified (SanityOnly). I was never able to reproduce this issue before the fix.
3.10.0-501.el7.x86_64
lvm2-2.02.165-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
lvm2-libs-2.02.165-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
lvm2-cluster-2.02.165-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
device-mapper-1.02.134-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
device-mapper-libs-1.02.134-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
device-mapper-event-1.02.134-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
device-mapper-event-libs-1.02.134-1.el7 BUILT: Wed Sep 7 11:04:22 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7 BUILT: Fri Jul 22 05:29:13 CDT 2016
[root@host-118 ~]# systemctl status lvm2-lvmetad
lvm2-lvmetad.service - LVM2 metadata daemon
Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2016-09-07 19:01:17 CDT; 1 day 22h ago
Docs: man:lvmetad(8)
Main PID: 472 (lvmetad)
CGroup: /system.slice/lvm2-lvmetad.service
/usr/sbin/lvmetad -f
creating lvm devices...
host-118: pvcreate /dev/sdf1 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdg1
host-118: vgcreate split_image /dev/sdf1 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sde1 /dev/sdg1
============================================================
Iteration 1 of 1 started at Fri Sep 9 16:45:25 CDT 2016
============================================================
SCENARIO - [split_w_tracking_io_merge]
Create a 3-way raid1 with fs data, verify data, split image with tracking, change data on raid vol, merge split image data back, verify origin data
host-118: lvcreate --type raid1 -m 2 -n split_tracking -L 1G split_image
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
Placing an ext filesystem on raid1 volume
mke2fs 1.42.9 (28-Dec-2013)
Mounting raid1 volume
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -n 500
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -v
Issuing a sync to force data to disk
splitting off leg from raid with tracking...
host-118: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking
LV VG Attr LSize Cpy%Sync Devices
split_tracking split_image rwi-aor--- 1.00g 100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
[split_tracking_rimage_0] split_image iwi-aor--- 1.00g /dev/sdf1(1)
[split_tracking_rimage_1] split_image iwi-aor--- 1.00g /dev/sda1(1)
split_tracking_rimage_2 split_image Iri-a-r--- 1.00g /dev/sdb1(1)
[split_tracking_rmeta_0] split_image ewi-aor--- 4.00m /dev/sdf1(0)
[split_tracking_rmeta_1] split_image ewi-aor--- 4.00m /dev/sda1(0)
[split_tracking_rmeta_2] split_image ewi-a-r--- 4.00m /dev/sdb1(0)
split_image-split_tracking: 0 2097152 raid raid1 3 AA 2097152/2097152 idle 0 0
+++ Mounting and verifying split image data +++
mount: /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.3276 -v
Writing new data to the raid and then merging back the split off image
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.3276 -n 500
Writing files to /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.3276 -n 500
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.3276 -v
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.3276 -v
Checking files on /mnt/split_tracking
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.3276 -v
Issuing a sync to force data to disk
Merge split off image split_image/split_tracking_rimage_2 back into the raid
lvconvert --merge split_image/split_tracking_rimage_2
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
LV VG Attr LSize Cpy%Sync Devices
split_tracking split_image rwi-aor--- 1.00g 100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
[split_tracking_rimage_0] split_image iwi-aor--- 1.00g /dev/sdf1(1)
[split_tracking_rimage_1] split_image iwi-aor--- 1.00g /dev/sda1(1)
[split_tracking_rimage_2] split_image iwi-aor--- 1.00g /dev/sdb1(1)
[split_tracking_rmeta_0] split_image ewi-aor--- 4.00m /dev/sdf1(0)
[split_tracking_rmeta_1] split_image ewi-aor--- 4.00m /dev/sda1(0)
[split_tracking_rmeta_2] split_image ewi-aor--- 4.00m /dev/sdb1(0)
split_image-split_tracking: 0 2097152 raid raid1 3 AAA 2097152/2097152 idle 0 0
Issuing a sync to force data to disk
AGAIN, splitting off leg from raid with tracking...
host-118: lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking
Waiting until all mirror|raid volumes become fully syncd...
1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec
LV VG Attr LSize Cpy%Sync Devices
split_tracking split_image rwi-aor--- 1.00g 100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
[split_tracking_rimage_0] split_image iwi-aor--- 1.00g /dev/sdf1(1)
[split_tracking_rimage_1] split_image iwi-aor--- 1.00g /dev/sda1(1)
split_tracking_rimage_2 split_image Iri-a-r--- 1.00g /dev/sdb1(1)
[split_tracking_rmeta_0] split_image ewi-aor--- 4.00m /dev/sdf1(0)
[split_tracking_rmeta_1] split_image ewi-aor--- 4.00m /dev/sda1(0)
[split_tracking_rmeta_2] split_image ewi-a-r--- 4.00m /dev/sdb1(0)
split_image-split_tracking: 0 2097152 raid raid1 3 AA 2097152/2097152 idle 0 0
+++ Mounting and verifying split image data +++
mount: /dev/mapper/split_image-split_tracking_rimage_2 is write-protected, mounting read-only
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.3276 -v
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.3276 -v
Checking files on /mnt/split_tracking_rimage_2
/usr/tests/sts-rhel7.3/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingC.3276 -v
Merge split off image split_image/split_tracking_rimage_2 back so it can be deleted
lvconvert --merge split_image/split_tracking_rimage_2
LV VG Attr LSize Cpy%Sync Devices
split_tracking split_image rwi-a-r--- 1.00g 100.00 split_tracking_rimage_0(0),split_tracking_rimage_1(0),split_tracking_rimage_2(0)
[split_tracking_rimage_0] split_image iwi-aor--- 1.00g /dev/sdf1(1)
[split_tracking_rimage_1] split_image iwi-aor--- 1.00g /dev/sda1(1)
[split_tracking_rimage_2] split_image iwi-aor--- 1.00g /dev/sdb1(1)
[split_tracking_rmeta_0] split_image ewi-aor--- 4.00m /dev/sdf1(0)
[split_tracking_rmeta_1] split_image ewi-aor--- 4.00m /dev/sda1(0)
[split_tracking_rmeta_2] split_image ewi-aor--- 4.00m /dev/sdb1(0)
split_image-split_tracking: 0 2097152 raid raid1 3 AAA 2097152/2097152 idle 0 0
Deactivating LV split_image/split_tracking on host-118... and removing
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1445.html |
Description of problem: splitting and merging a LV-RAID leg sometimes fails to update the leg properly. Nothing in logs indicating failure, just the data are not there. Version-Release number of selected component (if applicable): lvm2-2.02.105-14.el7.x86_64 kernel-3.10.0-121.el7.x86_64 (lvmetad is running, so far 0 reproducibility without lvmetad) How reproducible: low Steps to Reproduce: pvcreate /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 /dev/sdg1 /dev/sdj1 vgcreate split_image /dev/sdf1 /dev/sde1 /dev/sdb1 /dev/sdc1 /dev/sdg1 /dev/sdj1 pvs --noheadings lvcreate --type raid1 -m 2 -n split_tracking -L 1G split_image # WAIT FOR SYNC... dmsetup info mkfs /dev/split_image/split_tracking mkdir -p /mnt/split_tracking mount /dev/split_image/split_tracking /mnt/split_tracking # WRITE1 AND VERIFY: /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -n 500 /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -v sync lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking mkdir -p /mnt/split_tracking_rimage_2 mount /dev/split_image/split_tracking_rimage_2 /mnt/split_tracking_rimage_2 # VERIFY SPLIT LEG: /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.31800 -v umount /mnt/split_tracking_rimage_2 # WRITE TO ORIGIN AND VERIFY: /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.31800 -n 500 /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingB.31800 -v /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.31800 -n 500 /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingC.31800 -v /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking -f /tmp/split_trackingA.31800 -v sync lvconvert --merge split_image/split_tracking_rimage_2 # WAIT FOR SYNC... dmsetup info sync lvconvert --splitmirrors 1 --trackchanges split_image/split_tracking mkdir -p /mnt/split_tracking_rimage_2 mount /dev/split_image/split_tracking_rimage_2 /mnt/split_tracking_rimage_2 # VERIFY THE LEG WAS SYNCED: /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingA.31800 -v /usr/tests/sts-rhel7.0-testing/bin/checkit -w /mnt/split_tracking_rimage_2 -f /tmp/split_trackingB.31800 -v Actual results: The RAID LV contains correct data, but the leg was not synced. Expected results: All legs should be in sync. Additional info: