RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1618806 - RAID TAKEOVER: issues and corruption attempting raid5_n -> linear conversion
Summary: RAID TAKEOVER: issues and corruption attempting raid5_n -> linear conversion
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: LVM and device-mapper development team
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-17 16:16 UTC by Corey Marthaler
Modified: 2021-09-03 12:37 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-08-17 19:49:08 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Corey Marthaler 2018-08-17 16:16:04 UTC 
Description of problem:
Of all the hundreds of interim segtype conversion possibilities, this appears to be the only one calling for the user to use "--force", and ends up doing multiple takeovers to the existing segtype (raid5_n -> raid5_n -> raid5_n -> raid1 -> linear) and ends up causing corruption.


[root@hayes-01 ~]# lvs -a -o +devices,segtype
  LV                  VG         Attr       LSize   Cpy%Sync Devices                                                                             Type   
  lvol0               centipede2 -wi-a-----  20.00m          /dev/sdg1(236)                                                                      linear 
  lvol1               centipede2 -wi-a-----  20.00m          /dev/sdg1(241)                                                                      linear 
  lvol2               centipede2 -wi-a-----  20.00m          /dev/sdh1(236)                                                                      linear 
  lvol3               centipede2 -wi-a-----  20.00m          /dev/sdh1(241)                                                                      linear 
  lvol4               centipede2 -wi-a-----  20.00m          /dev/sdi1(236)                                                                      linear 
  lvol5               centipede2 -wi-a-----  20.00m          /dev/sdi1(241)                                                                      linear 
  lvol6               centipede2 -wi-a-----  20.00m          /dev/sdk1(236)                                                                      linear 
  lvol7               centipede2 -wi-a-----  20.00m          /dev/sdk1(241)                                                                      linear 
  takeover            centipede2 rwi-aor---   2.81g 100.00   takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0) raid5_n
  [takeover_rimage_0] centipede2 iwi-aor--- 960.00m          /dev/sdg1(1)                                                                        linear 
  [takeover_rimage_0] centipede2 iwi-aor--- 960.00m          /dev/sdg1(246)                                                                      linear 
  [takeover_rimage_1] centipede2 iwi-aor--- 960.00m          /dev/sdk1(1)                                                                        linear 
  [takeover_rimage_1] centipede2 iwi-aor--- 960.00m          /dev/sdk1(246)                                                                      linear 
  [takeover_rimage_2] centipede2 iwi-aor--- 960.00m          /dev/sdi1(1)                                                                        linear 
  [takeover_rimage_2] centipede2 iwi-aor--- 960.00m          /dev/sdi1(246)                                                                      linear 
  [takeover_rimage_3] centipede2 iwi-aor--- 960.00m          /dev/sdh1(1)                                                                        linear 
  [takeover_rimage_3] centipede2 iwi-aor--- 960.00m          /dev/sdh1(246)                                                                      linear 
  [takeover_rmeta_0]  centipede2 ewi-aor---   4.00m          /dev/sdg1(0)                                                                        linear 
  [takeover_rmeta_1]  centipede2 ewi-aor---   4.00m          /dev/sdk1(0)                                                                        linear 
  [takeover_rmeta_2]  centipede2 ewi-aor---   4.00m          /dev/sdi1(0)                                                                        linear 
  [takeover_rmeta_3]  centipede2 ewi-aor---   4.00m          /dev/sdh1(0)                                                                        linear 

[root@hayes-01 ~]# lvconvert --yes   --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active and open logical volume centipede2/takeover will shrink it from 2.81 GiB to 960.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  If that leaves the logical volume larger than 2160 extents due to stripe rounding,
  you may want to grow the content afterwards (filesystem etc.)
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/takeover"
  Can't remove stripes without --force option.
  Reshape request failed on LV centipede2/takeover.

[root@hayes-01 ~]# lvconvert --yes --force  --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active and open logical volume centipede2/takeover will shrink it from 2.81 GiB to 960.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  If that leaves the logical volume larger than 2160 extents due to stripe rounding,
  you may want to grow the content afterwards (filesystem etc.)
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/takeover"
  Logical volume centipede2/takeover successfully converted.

[root@hayes-01 ~]# lvs -a -o +devices,segtype
  LV                  VG         Attr       LSize   Cpy%Sync Devices                                                                             Type   
  lvol0               centipede2 -wi-a-----  20.00m          /dev/sdg1(236)                                                                      linear 
  lvol1               centipede2 -wi-a-----  20.00m          /dev/sdg1(241)                                                                      linear 
  lvol2               centipede2 -wi-a-----  20.00m          /dev/sdh1(236)                                                                      linear 
  lvol3               centipede2 -wi-a-----  20.00m          /dev/sdh1(241)                                                                      linear 
  lvol4               centipede2 -wi-a-----  20.00m          /dev/sdi1(236)                                                                      linear 
  lvol5               centipede2 -wi-a-----  20.00m          /dev/sdi1(241)                                                                      linear 
  lvol6               centipede2 -wi-a-----  20.00m          /dev/sdk1(236)                                                                      linear 
  lvol7               centipede2 -wi-a-----  20.00m          /dev/sdk1(241)                                                                      linear 
  takeover            centipede2 rwi-aor-s-   2.81g 56.73    takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0) raid5_n
  [takeover_rimage_0] centipede2 Iwi-aor--- 964.00m          /dev/sdg1(1)                                                                        linear 
  [takeover_rimage_0] centipede2 Iwi-aor--- 964.00m          /dev/sdg1(246)                                                                      linear 
  [takeover_rimage_1] centipede2 Iwi-aor--- 964.00m          /dev/sdk1(1)                                                                        linear 
  [takeover_rimage_1] centipede2 Iwi-aor--- 964.00m          /dev/sdk1(246)                                                                      linear 
  [takeover_rimage_2] centipede2 Iwi-aor-R- 964.00m          /dev/sdi1(1)                                                                        linear 
  [takeover_rimage_2] centipede2 Iwi-aor-R- 964.00m          /dev/sdi1(246)                                                                      linear 
  [takeover_rimage_3] centipede2 Iwi-aor-R- 964.00m          /dev/sdh1(1)                                                                        linear 
  [takeover_rimage_3] centipede2 Iwi-aor-R- 964.00m          /dev/sdh1(246)                                                                      linear 
  [takeover_rmeta_0]  centipede2 ewi-aor---   4.00m          /dev/sdg1(0)                                                                        linear 
  [takeover_rmeta_1]  centipede2 ewi-aor---   4.00m          /dev/sdk1(0)                                                                        linear 
  [takeover_rmeta_2]  centipede2 ewi-aor-R-   4.00m          /dev/sdi1(0)                                                                        linear 
  [takeover_rmeta_3]  centipede2 ewi-aor-R-   4.00m          /dev/sdh1(0)                                                                        linear 

[root@hayes-01 ~]# lvs -a -o +devices,segtype
  LV                  VG         Attr       LSize   Cpy%Sync Devices                                                                             Type   
  lvol0               centipede2 -wi-a-----  20.00m          /dev/sdg1(236)                                                                      linear 
  lvol1               centipede2 -wi-a-----  20.00m          /dev/sdg1(241)                                                                      linear 
  lvol2               centipede2 -wi-a-----  20.00m          /dev/sdh1(236)                                                                      linear 
  lvol3               centipede2 -wi-a-----  20.00m          /dev/sdh1(241)                                                                      linear 
  lvol4               centipede2 -wi-a-----  20.00m          /dev/sdi1(236)                                                                      linear 
  lvol5               centipede2 -wi-a-----  20.00m          /dev/sdi1(241)                                                                      linear 
  lvol6               centipede2 -wi-a-----  20.00m          /dev/sdk1(236)                                                                      linear 
  lvol7               centipede2 -wi-a-----  20.00m          /dev/sdk1(241)                                                                      linear 
  takeover            centipede2 rwi-aor-R-   2.81g 100.00   takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0) raid5_n
  [takeover_rimage_0] centipede2 iwi-aor--- 964.00m          /dev/sdg1(1)                                                                        linear 
  [takeover_rimage_0] centipede2 iwi-aor--- 964.00m          /dev/sdg1(246)                                                                      linear 
  [takeover_rimage_1] centipede2 iwi-aor--- 964.00m          /dev/sdk1(1)                                                                        linear 
  [takeover_rimage_1] centipede2 iwi-aor--- 964.00m          /dev/sdk1(246)                                                                      linear 
  [takeover_rimage_2] centipede2 Iwi-aor-R- 964.00m          /dev/sdi1(1)                                                                        linear 
  [takeover_rimage_2] centipede2 Iwi-aor-R- 964.00m          /dev/sdi1(246)                                                                      linear 
  [takeover_rimage_3] centipede2 Iwi-aor-R- 964.00m          /dev/sdh1(1)                                                                        linear 
  [takeover_rimage_3] centipede2 Iwi-aor-R- 964.00m          /dev/sdh1(246)                                                                      linear 
  [takeover_rmeta_0]  centipede2 ewi-aor---   4.00m          /dev/sdg1(0)                                                                        linear 
  [takeover_rmeta_1]  centipede2 ewi-aor---   4.00m          /dev/sdk1(0)                                                                        linear 
  [takeover_rmeta_2]  centipede2 ewi-aor-R-   4.00m          /dev/sdi1(0)                                                                        linear 
  [takeover_rmeta_3]  centipede2 ewi-aor-R-   4.00m          /dev/sdh1(0)                                                                        linear 

[root@hayes-01 ~]# lvconvert --yes   --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  Logical volume centipede2/takeover successfully converted.

[root@hayes-01 ~]# lvconvert --yes   --type linear centipede2/takeover
  Replaced LV type linear with possible type raid1.
  Repeat this command to convert to linear after an interim conversion has finished.
  Logical volume centipede2/takeover successfully converted.

[root@hayes-01 ~]# lvconvert --yes   --type linear centipede2/takeover
  Logical volume centipede2/takeover successfully converted.

[root@hayes-01 ~]# lvs -a -o +devices,segtype
  LV       VG         Attr       LSize   Cpy%Sync Devices        Type  
  lvol0    centipede2 -wi-a-----  20.00m          /dev/sdg1(236) linear
  lvol1    centipede2 -wi-a-----  20.00m          /dev/sdg1(241) linear
  lvol2    centipede2 -wi-a-----  20.00m          /dev/sdh1(236) linear
  lvol3    centipede2 -wi-a-----  20.00m          /dev/sdh1(241) linear
  lvol4    centipede2 -wi-a-----  20.00m          /dev/sdi1(236) linear
  lvol5    centipede2 -wi-a-----  20.00m          /dev/sdi1(241) linear
  lvol6    centipede2 -wi-a-----  20.00m          /dev/sdk1(236) linear
  lvol7    centipede2 -wi-a-----  20.00m          /dev/sdk1(241) linear
  takeover centipede2 -wi-ao---- 960.00m          /dev/sdg1(2)   linear
  takeover centipede2 -wi-ao---- 960.00m          /dev/sdg1(246) linear

[root@hayes-01 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/centipede2-takeover  2.9G   60M  2.8G   3% /mnt/takeover

# The filesystem is corrupted
[root@hayes-01 ~]# ls -lrt /mnt/takeover 
ls: cannot access /mnt/takeover: Input/output error

[root@hayes-01 ~]# umount /mnt/*

[root@hayes-01 ~]# mount /dev/centipede2/takeover /mnt/takeover
mount: /dev/mapper/centipede2-takeover: can't read superblock


Version-Release number of selected component (if applicable):
3.10.0-931.el7.x86_64

lvm2-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-libs-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-cluster-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-lockd-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
lvm2-python-boom-0.9-5.el7    BUILT: Wed Aug  1 11:24:13 CDT 2018
cmirror-2.02.180-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-libs-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-event-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-event-libs-1.02.149-2.el7    BUILT: Wed Aug  1 11:22:48 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017


How reproducible:
Everytime
Comment 2 Corey Marthaler 2018-08-17 16:22:36 UTC 
Here's the actual test output. It appears the conversion to raid1 is what causes the corruption.

================================================================================
Iteration 0.1 started at Fri Aug 17 10:55:18 CDT 2018
================================================================================
Scenario random_interim_conversion: Convert Random striped type to indirect interim volume
********* Take over hash info for this scenario *********
* from type:    raid5_n
* to type:      linear
* from legs:    3
* to legs:      3
* from region:  1024.00k
* to region:    0
* contiguous:   0
******************************************************

Creating original volume on hayes-01...
hayes-01: lvcreate  --type raid5_n -R 1024.00k -i 3 -n takeover -L 2.75G centipede2
Waiting until all mirror|raid volumes become fully syncd...
   0/1 mirror(s) are fully synced: ( 49.97% )
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Placing a spacer on all raid image PVs so that expansion will have to be placed beyond
Extending raid beyond spacer
        lvextend -L +50M centipede2/takeover

Current volume device structure:
  LV                  Type    Attr       LSize   Cpy%Sync Devices
  lvol0               linear  -wi-a-----  20.00m          /dev/sdg1(236)
  lvol1               linear  -wi-a-----  20.00m          /dev/sdg1(241)
  lvol2               linear  -wi-a-----  20.00m          /dev/sdh1(236)
  lvol3               linear  -wi-a-----  20.00m          /dev/sdh1(241)
  lvol4               linear  -wi-a-----  20.00m          /dev/sdi1(236)
  lvol5               linear  -wi-a-----  20.00m          /dev/sdi1(241)
  lvol6               linear  -wi-a-----  20.00m          /dev/sdk1(236)
  lvol7               linear  -wi-a-----  20.00m          /dev/sdk1(241)
  takeover            raid5_n rwi-a-r---   2.81g 97.92    takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0)
  [takeover_rimage_0] linear  Iwi-aor--- 960.00m          /dev/sdg1(1)
  [takeover_rimage_0] linear  Iwi-aor--- 960.00m          /dev/sdg1(246)
  [takeover_rimage_1] linear  Iwi-aor--- 960.00m          /dev/sdk1(1)
  [takeover_rimage_1] linear  Iwi-aor--- 960.00m          /dev/sdk1(246)
  [takeover_rimage_2] linear  Iwi-aor--- 960.00m          /dev/sdi1(1)
  [takeover_rimage_2] linear  Iwi-aor--- 960.00m          /dev/sdi1(246)
  [takeover_rimage_3] linear  Iwi-aor--- 960.00m          /dev/sdh1(1)
  [takeover_rimage_3] linear  Iwi-aor--- 960.00m          /dev/sdh1(246)
  [takeover_rmeta_0]  linear  ewi-aor---   4.00m          /dev/sdg1(0)
  [takeover_rmeta_1]  linear  ewi-aor---   4.00m          /dev/sdk1(0)
  [takeover_rmeta_2]  linear  ewi-aor---   4.00m          /dev/sdi1(0)
  [takeover_rmeta_3]  linear  ewi-aor---   4.00m          /dev/sdh1(0)

Creating xfs on top of mirror(s) on hayes-01...
Mounting mirrored xfs filesystems on hayes-01...

Writing verification files (checkit) to mirror(s) on...
        ---- hayes-01 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----

TAKEOVER: lvconvert --force --yes   --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active and open logical volume centipede2/takeover will shrink it from 2.81 GiB to 960.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/takeover"

Current volume device structure:
  LV                  Type    Attr       LSize   Cpy%Sync Devices
  lvol0               linear  -wi-a-----  20.00m          /dev/sdg1(236)
  lvol1               linear  -wi-a-----  20.00m          /dev/sdg1(241)
  lvol2               linear  -wi-a-----  20.00m          /dev/sdh1(236)
  lvol3               linear  -wi-a-----  20.00m          /dev/sdh1(241)
  lvol4               linear  -wi-a-----  20.00m          /dev/sdi1(236)
  lvol5               linear  -wi-a-----  20.00m          /dev/sdi1(241)
  lvol6               linear  -wi-a-----  20.00m          /dev/sdk1(236)
  lvol7               linear  -wi-a-----  20.00m          /dev/sdk1(241)
  takeover            raid5_n rwi-aor-s-   2.81g 56.67    takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0)
  [takeover_rimage_0] linear  Iwi-aor--- 964.00m          /dev/sdg1(1)
  [takeover_rimage_0] linear  Iwi-aor--- 964.00m          /dev/sdg1(246)
  [takeover_rimage_1] linear  Iwi-aor--- 964.00m          /dev/sdk1(1)
  [takeover_rimage_1] linear  Iwi-aor--- 964.00m          /dev/sdk1(246)
  [takeover_rimage_2] linear  Iwi-aor-R- 964.00m          /dev/sdi1(1)
  [takeover_rimage_2] linear  Iwi-aor-R- 964.00m          /dev/sdi1(246)
  [takeover_rimage_3] linear  Iwi-aor-R- 964.00m          /dev/sdh1(1)
  [takeover_rimage_3] linear  Iwi-aor-R- 964.00m          /dev/sdh1(246)
  [takeover_rmeta_0]  linear  ewi-aor---   4.00m          /dev/sdg1(0)
  [takeover_rmeta_1]  linear  ewi-aor---   4.00m          /dev/sdk1(0)
  [takeover_rmeta_2]  linear  ewi-aor-R-   4.00m          /dev/sdi1(0)
  [takeover_rmeta_3]  linear  ewi-aor-R-   4.00m          /dev/sdh1(0)

Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----

current segtype doesn't match the takeover attempted
current: raid5_n ne expected: linear
Continuing to run this same takeover cmd until at the requested segtype (attempts 1)
TAKEOVER: lvconvert --force --yes   --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.

Current volume device structure:
  LV                  Type    Attr       LSize   Cpy%Sync Devices
  lvol0               linear  -wi-a-----  20.00m          /dev/sdg1(236)
  lvol1               linear  -wi-a-----  20.00m          /dev/sdg1(241)
  lvol2               linear  -wi-a-----  20.00m          /dev/sdh1(236)
  lvol3               linear  -wi-a-----  20.00m          /dev/sdh1(241)
  lvol4               linear  -wi-a-----  20.00m          /dev/sdi1(236)
  lvol5               linear  -wi-a-----  20.00m          /dev/sdi1(241)
  lvol6               linear  -wi-a-----  20.00m          /dev/sdk1(236)
  lvol7               linear  -wi-a-----  20.00m          /dev/sdk1(241)
  takeover            raid5_n rwi-aor--- 960.00m 100.00   takeover_rimage_0(0),takeover_rimage_1(0)
  [takeover_rimage_0] linear  iwi-aor--- 964.00m          /dev/sdg1(1)
  [takeover_rimage_0] linear  iwi-aor--- 964.00m          /dev/sdg1(246)
  [takeover_rimage_1] linear  iwi-aor--- 964.00m          /dev/sdk1(1)
  [takeover_rimage_1] linear  iwi-aor--- 964.00m          /dev/sdk1(246)
  [takeover_rmeta_0]  linear  ewi-aor---   4.00m          /dev/sdg1(0)
  [takeover_rmeta_1]  linear  ewi-aor---   4.00m          /dev/sdk1(0)


Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----

current segtype doesn't match the takeover attempted
current: raid5_n ne expected: linear
Continuing to run this same takeover cmd until at the requested segtype (attempts 2)


# It appears to be this conversion that causes the corruption:
TAKEOVER: lvconvert --force --yes   --type linear centipede2/takeover
  Replaced LV type linear with possible type raid1.
  Repeat this command to convert to linear after an interim conversion has finished.

Current volume device structure:
  LV                  Type   Attr       LSize   Cpy%Sync Devices
  lvol0               linear -wi-a-----  20.00m          /dev/sdg1(236)
  lvol1               linear -wi-a-----  20.00m          /dev/sdg1(241)
  lvol2               linear -wi-a-----  20.00m          /dev/sdh1(236)
  lvol3               linear -wi-a-----  20.00m          /dev/sdh1(241)
  lvol4               linear -wi-a-----  20.00m          /dev/sdi1(236)
  lvol5               linear -wi-a-----  20.00m          /dev/sdi1(241)
  lvol6               linear -wi-a-----  20.00m          /dev/sdk1(236)
  lvol7               linear -wi-a-----  20.00m          /dev/sdk1(241)
  takeover            raid1  rwi-aor--- 960.00m 100.00   takeover_rimage_0(0),takeover_rimage_1(0)
  [takeover_rimage_0] linear iwi-aor--- 960.00m          /dev/sdg1(2)
  [takeover_rimage_0] linear iwi-aor--- 960.00m          /dev/sdg1(246)
  [takeover_rimage_1] linear iwi-aor--- 960.00m          /dev/sdk1(2)
  [takeover_rimage_1] linear iwi-aor--- 960.00m          /dev/sdk1(246)
  [takeover_rmeta_0]  linear ewi-aor---   4.00m          /dev/sdg1(0)
  [takeover_rmeta_1]  linear ewi-aor---   4.00m          /dev/sdk1(0)


Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----
Can not stat bnpiblpnmfqywmenh: Input/output error
checkit write verify failed
Corruption was **NOT** expected for this scenario!
Comment 3 Corey Marthaler 2018-08-17 18:34:08 UTC 
Actually, it appears the corruption can happen after the first takeover attempt.

================================================================================
Iteration 0.1 started at Fri Aug 17 13:21:38 CDT 2018
================================================================================
Scenario random_interim_conversion: Convert Random striped type to indirect interim volume
********* Take over hash info for this scenario *********
* from type:    raid5_n
* to type:      linear
* from legs:    3
* to legs:      3
* from region:  8192.00k
* to region:    0
* contiguous:   1
******************************************************

Creating original volume on hayes-01...
hayes-01: lvcreate  --type raid5_n -R 8192.00k -i 3 -n takeover -L 2.75G centipede2
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Current volume device structure:
  LV                  Type    Attr       LSize   Cpy%Sync Devices
  takeover            raid5_n rwi-a-r---   2.75g 100.00   takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0)
  [takeover_rimage_0] linear  iwi-aor--- 940.00m          /dev/sdg1(1)
  [takeover_rimage_1] linear  iwi-aor--- 940.00m          /dev/sdk1(1)
  [takeover_rimage_2] linear  iwi-aor--- 940.00m          /dev/sdi1(1)
  [takeover_rimage_3] linear  iwi-aor--- 940.00m          /dev/sdh1(1)
  [takeover_rmeta_0]  linear  ewi-aor---   4.00m          /dev/sdg1(0)
  [takeover_rmeta_1]  linear  ewi-aor---   4.00m          /dev/sdk1(0)
  [takeover_rmeta_2]  linear  ewi-aor---   4.00m          /dev/sdi1(0)
  [takeover_rmeta_3]  linear  ewi-aor---   4.00m          /dev/sdh1(0)

Creating xfs on top of mirror(s) on hayes-01...
Mounting mirrored xfs filesystems on hayes-01...

Writing verification files (checkit) to mirror(s) on...
        ---- hayes-01 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----

TAKEOVER: lvconvert --force --yes   --type linear centipede2/takeover
  Converting raid5_n LV centipede2/takeover to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active and open logical volume centipede2/takeover will shrink it from 2.75 GiB to 940.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/takeover"
Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )

<fail name="hayes-01_takeover"  pid="21472" time="Fri Aug 17 13:23:14 2018 -0500" type="cmd" duration="60" ec="1" />

Sleeping 15 sec
Current volume device structure:
  LV                  Type    Attr       LSize   Cpy%Sync Devices                                                                            
  takeover            raid5_n rwi-aor-R-   2.75g 100.00   takeover_rimage_0(0),takeover_rimage_1(0),takeover_rimage_2(0),takeover_rimage_3(0)
  [takeover_rimage_0] linear  iwi-aor--- 944.00m          /dev/sdg1(1)                                                                       
  [takeover_rimage_1] linear  iwi-aor--- 944.00m          /dev/sdk1(1)                                                                       
  [takeover_rimage_2] linear  Iwi-aor-R- 944.00m          /dev/sdi1(1)                                                                       
  [takeover_rimage_3] linear  Iwi-aor-R- 944.00m          /dev/sdh1(1)                                                                       
  [takeover_rmeta_0]  linear  ewi-aor---   4.00m          /dev/sdg1(0)                                                                       
  [takeover_rmeta_1]  linear  ewi-aor---   4.00m          /dev/sdk1(0)                                                                       
  [takeover_rmeta_2]  linear  ewi-aor-R-   4.00m          /dev/sdi1(0)                                                                       
  [takeover_rmeta_3]  linear  ewi-aor-R-   4.00m          /dev/sdh1(0)                                                                       

Verifying files (checkit) on mirror(s) on...
        ---- hayes-01 ----
Can not cd to /mnt/takeover/checkit: Input/output error
checkit write verify failed




Aug 17 13:22:45 hayes-01 qarshd[54963]: Running cmdline: lvconvert --force --yes --type linear centipede2/takeover
Aug 17 13:22:45 hayes-01 dmeventd[70059]: No longer monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:45 hayes-01 lvm[70059]: Monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:45 hayes-01 kernel: md/raid:mdX: device dm-1 operational as raid disk 0
Aug 17 13:22:45 hayes-01 kernel: md/raid:mdX: device dm-3 operational as raid disk 1
Aug 17 13:22:45 hayes-01 kernel: md/raid:mdX: device dm-5 operational as raid disk 2
Aug 17 13:22:45 hayes-01 kernel: md/raid:mdX: device dm-7 operational as raid disk 3
Aug 17 13:22:45 hayes-01 kernel: md/raid:mdX: raid level 5 active with 4 out of 4 devices, algorithm 5
Aug 17 13:22:45 hayes-01 dmeventd[70059]: No longer monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:45 hayes-01 kernel: dm-8: detected capacity change from 2956984320 to 985661440
Aug 17 13:22:45 hayes-01 kernel: VFS: busy inodes on changed media or resized disk dm-8
Aug 17 13:22:45 hayes-01 lvm[70059]: Monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:45 hayes-01 lvm[70059]: raid5_n array, centipede2-takeover, is not in-sync.
Aug 17 13:22:45 hayes-01 kernel: md: reshape of RAID array mdX
Aug 17 13:22:57 hayes-01 kernel: md/raid:mdX: device dm-1 operational as raid disk 0
Aug 17 13:22:57 hayes-01 kernel: md/raid:mdX: device dm-3 operational as raid disk 1
Aug 17 13:22:57 hayes-01 kernel: md/raid:mdX: device dm-5 operational as raid disk 2
Aug 17 13:22:57 hayes-01 kernel: md/raid:mdX: device dm-7 operational as raid disk 3
Aug 17 13:22:57 hayes-01 kernel: md/raid:mdX: raid level 5 active with 2 out of 2 devices, algorithm 5
Aug 17 13:22:57 hayes-01 dmeventd[70059]: No longer monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:57 hayes-01 kernel: md: mdX: reshape interrupted.
Aug 17 13:22:57 hayes-01 kernel: dm-8: detected capacity change from 2956984320 to 985661440
Aug 17 13:22:57 hayes-01 kernel: VFS: busy inodes on changed media or resized disk dm-8
Aug 17 13:22:57 hayes-01 lvm[70059]: Monitoring RAID device centipede2-takeover for events.
Aug 17 13:22:57 hayes-01 lvm[70059]: raid5_n array, centipede2-takeover, is not in-sync.
Aug 17 13:22:57 hayes-01 kernel: md: reshape of RAID array mdX
Aug 17 13:23:05 hayes-01 kernel: md: mdX: reshape done.
Aug 17 13:23:05 hayes-01 lvm[70059]: raid5_n array, centipede2-takeover, is not in-sync.
[...]
Aug 17 13:23:14 hayes-01 kernel: attempt to access beyond end of device
Aug 17 13:23:14 hayes-01 kernel: dm-8: rw=7217, want=2887680, limit=1925120
Aug 17 13:23:14 hayes-01 kernel: XFS (dm-8): metadata I/O error: block 0x2c0f80 ("xlog_iodone") error 5 numblks 128
Aug 17 13:23:14 hayes-01 kernel: XFS (dm-8): xfs_do_force_shutdown(0x2) called from line 1221 of file fs/xfs/xfs_log.c.  Return address = 0xffffffffc0665270
Aug 17 13:23:14 hayes-01 kernel: XFS (dm-8): Log I/O Error Detected.  Shutting down filesystem
Aug 17 13:23:14 hayes-01 kernel: XFS (dm-8): Please umount the filesystem and rectify the problem(s)
Comment 4 Corey Marthaler 2018-08-17 19:49:08 UTC 
Originally, I thought this corruption was a bug because it wasn't like other reshape scenarios where you specifically specify a reduction in images from N -> N-1 stripes, but instead lvm does it "automatically".

But after re-evaluating, lvm does A. provide a warning about corruption B. require a --force and C. then prompt the user w/ a --yes. So, closing NOTABUG

[root@host-086 ~]# lvconvert  --type linear centipede2/lvol0
  Converting raid5_n LV centipede2/lvol0 to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active logical volume centipede2/lvol0 will shrink it from 108.00 MiB to 36.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  Interrupt the conversion and run "lvresize -y -l81 centipede2/lvol0" to keep the current size if not done already!
  If that leaves the logical volume larger than 81 extents due to stripe rounding,
  you may want to grow the content afterwards (filesystem etc.)
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/lvol0"
  Can't remove stripes without --force option.
  Reshape request failed on LV centipede2/lvol0.


[root@host-086 ~]# lvconvert --force  --type linear centipede2/lvol0
  Converting raid5_n LV centipede2/lvol0 to 2 stripes first.
  Replaced LV type linear with possible type raid5_n.
  Repeat this command to convert to linear after an interim conversion has finished.
  WARNING: Removing stripes from active logical volume centipede2/lvol0 will shrink it from 108.00 MiB to 36.00 MiB!
  THIS MAY DESTROY (PARTS OF) YOUR DATA!
  Interrupt the conversion and run "lvresize -y -l81 centipede2/lvol0" to keep the current size if not done already!
  If that leaves the logical volume larger than 81 extents due to stripe rounding,
  you may want to grow the content afterwards (filesystem etc.)
  WARNING: to remove freed stripes after the conversion has finished, you have to run "lvconvert --stripes 1 centipede2/lvol0"
Are you sure you want to remove 2 images from raid5_n LV centipede2/lvol0? [y/n]:
Note You need to log in before you can comment on or make changes to this bug.