RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1712983 - "Internal error: LV segments corrupted" during syncing disk -> mirrored log conversion with supplied -R regionsize [rhel-7.9.z]
Summary: "Internal error: LV segments corrupted" during syncing disk -> mirrored log c...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.7
Hardware: x86_64
OS: Linux
urgent
unspecified
Target Milestone: rc
: ---
Assignee: Heinz Mauelshagen
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-22 15:53 UTC by Corey Marthaler
Modified: 2021-09-03 12:55 UTC (History)
8 users (show)

Fixed In Version: lvm2-2.02.187-6.el7_9.1.src.rpm
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 16:09:11 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
verbose lvconvert attempt (35.73 KB, text/plain)
2019-05-22 16:00 UTC, Corey Marthaler 		no flags Details
View All

Description Corey Marthaler 2019-05-22 15:53:17 UTC 
Description of problem:
[root@hayes-01 ~]# pvcreate /dev/sdj1 /dev/sde1 /dev/sdd1 /dev/sdb1 /dev/sdg1 /dev/sdh1 /dev/sdf1 /dev/sdc1 /dev/sdk1 /dev/sdi1
  Physical volume "/dev/sdj1" successfully created.
  Physical volume "/dev/sde1" successfully created.
  Physical volume "/dev/sdd1" successfully created.
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdg1" successfully created.
  Physical volume "/dev/sdh1" successfully created.
  Physical volume "/dev/sdf1" successfully created.
  Physical volume "/dev/sdc1" successfully created.
  Physical volume "/dev/sdk1" successfully created.
  Physical volume "/dev/sdi1" successfully created.
[root@hayes-01 ~]# vgcreate   mirror_sanity /dev/sdj1 /dev/sde1 /dev/sdd1 /dev/sdb1 /dev/sdg1 /dev/sdh1 /dev/sdf1 /dev/sdc1 /dev/sdk1 /dev/sdi1
  Volume group "mirror_sanity" successfully created
[root@hayes-01 ~]# lvcreate  --type mirror -m 1 -n mirror_log_convert --mirrorlog core -l 524465 -R 16 mirror_sanity /dev/sdk1 /dev/sdj1 /dev/sde1 /dev/sdd1
  Logical volume "mirror_log_convert" created.
[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG            Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  mirror_log_convert            mirror_sanity mwi-a-m--- 2.00t                                    0.02             mirror_log_convert_mimage_0(0),mirror_log_convert_mimage_1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdj1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdk1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sde1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdd1(0)
[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG            Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  mirror_log_convert            mirror_sanity mwi-a-m--- 2.00t                                    0.45             mirror_log_convert_mimage_0(0),mirror_log_convert_mimage_1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdj1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdk1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sde1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                     /dev/sdd1(0)
[root@hayes-01 ~]# lvconvert --type mirror -m 1 --mirrorlog disk /dev/mirror_sanity/mirror_log_convert
  Logical volume mirror_sanity/mirror_log_convert converted.
[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG            Attr       LSize Pool Origin Data%  Meta%  Move Log                       Cpy%Sync Convert Devices
  mirror_log_convert            mirror_sanity mwi-a-m--- 2.00t                                [mirror_log_convert_mlog] 0.03             mirror_log_convert_mimage_0(0),mirror_log_convert_mimage_1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdj1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdk1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sde1(0)
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdd1(0)
  [mirror_log_convert_mlog]     mirror_sanity lwi-aom--- 4.00m                                                                           /dev/sdi1(0)
[root@hayes-01 ~]# lvconvert --type mirror -m 1 --mirrorlog mirrored /dev/mirror_sanity/mirror_log_convert
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  LV mirror_log_convert_mlog, segment 1 invalid: region size is bigger then LV itself for mirror segment.
  Internal error: LV segments corrupted in mirror_log_convert_mlog.
[root@hayes-01 ~]# echo $?
5

[root@hayes-01 ~]# dmsetup status
mirror_sanity-mirror_log_convert_mlog: 0 8192 linear
mirror_sanity-mirror_log_convert_mimage_1: 0 3905937408 linear
mirror_sanity-mirror_log_convert_mimage_1: 3905937408 390479872 linear
mirror_sanity-mirror_log_convert_mimage_0: 0 3905937408 linear
mirror_sanity-mirror_log_convert_mimage_0: 3905937408 390479872 linear
mirror_sanity-mirror_log_convert: 0 4296417280 mirror 2 253:0 253:1 4367/131117 1 AA 3 disk 253:3 A


Version-Release number of selected component (if applicable):
3.10.0-1048.el7.x86_64

lvm2-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-libs-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-libs-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-libs-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-persistent-data-0.8.1-1.el7    BUILT: Sat May  4 14:53:53 CDT 2019
Comment 2 Corey Marthaler 2019-05-22 16:00:46 UTC 
Created attachment 1572095 [details]
verbose lvconvert attempt
Comment 3 Corey Marthaler 2019-05-22 16:45:18 UTC 
This is *not* a regression and apparently existed in RHEL7.6 as well.

[root@hayes-01 ~]# lvcreate  --type mirror -m 1 -n mirror_log_convert --mirrorlog core -l 524465 -R 16 mirror_sanity /dev/sdk1 /dev/sdj1 /dev/sde1 /dev/sdd1
  Logical volume "mirror_log_convert" created.
[root@hayes-01 ~]# lvconvert --type mirror -m 1 --mirrorlog disk /dev/mirror_sanity/mirror_log_convert
  Logical volume mirror_sanity/mirror_log_convert converted.
[root@hayes-01 ~]# lvs -a -o +devices
  LV                            VG            Attr       LSize Pool Origin Data%  Meta%  Move Log                       Cpy%Sync Convert Devices                                                      
  mirror_log_convert            mirror_sanity mwi-a-m--- 2.00t                                [mirror_log_convert_mlog] 0.02             mirror_log_convert_mimage_0(0),mirror_log_convert_mimage_1(0)
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdj1(0)                                                 
  [mirror_log_convert_mimage_0] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdk1(0)                                                 
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sde1(0)                                                 
  [mirror_log_convert_mimage_1] mirror_sanity Iwi-aom--- 2.00t                                                                           /dev/sdd1(0)                                                 
  [mirror_log_convert_mlog]     mirror_sanity lwi-aom--- 4.00m                                                                           /dev/sdh1(0)                                                 
[root@hayes-01 ~]# lvconvert --type mirror -m 1 --mirrorlog mirrored /dev/mirror_sanity/mirror_log_convert
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  LV mirror_log_convert_mlog, segment 1 invalid: region size is bigger then LV itself for mirror segment.
  Internal error: LV segments corrupted in mirror_log_convert_mlog.


3.10.0-957.el7.x86_64

lvm2-2.02.180-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
lvm2-libs-2.02.180-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
package lvm2-cluster is not installed
device-mapper-1.02.149-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
device-mapper-libs-1.02.149-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
device-mapper-event-1.02.149-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
device-mapper-event-libs-1.02.149-8.el7    BUILT: Mon Sep 10 04:45:22 CDT 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 05:07:18 CST 2017
Comment 4 Corey Marthaler 2019-05-28 20:47:55 UTC 
It appears the the -R regionsize in the original create is key to reproducing this issue.
Comment 5 Heinz Mauelshagen 2020-07-06 16:36:35 UTC 
Corey,

works here with the following el7 versions.
Please reopen in case of reproducer.

# lvm version
  LVM version:     2.02.185(2)-RHEL7 (2019-05-13)
  Library version: 1.02.158-RHEL7 (2019-05-13)
  Driver version:  4.37.1
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=
/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/l
ib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm
 --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with
-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --e
nable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvml
ockd-sanlock --enable-dmfilemapd

# uname -r
3.10.0-1062.el7.x86_64

# vgcreate  t /dev/sd*
  Volume group "t" successfully created

# vgs t
  VG #PV #LV #SN Attr   VSize  VFree 
  t   64   0   0 wz--n- 63.75g 63.75g

# lvcreate --ty mirror -m1 -nm --mirrorlog core -L2g -R 16 t
  Logical volume "m" created.
          

# lvs t
  LV   VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  m    t          mwi-a-m---   2.00g                                    100.00          

# lvconvert -m1 --mirrorlog mirrored t/m
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  Logical volume t/m converted.

# lvs -ao+devices t
  LV                VG Attr       LSize Pool Origin Data%  Meta%  Move Log      Cpy%Sync Convert Devices                              
  m                 t  mwi-a-m--- 2.00g                                [m_mlog] 100.00           m_mimage_0(0),m_mimage_1(0)          
  [m_mimage_0]      t  iwi-aom--- 2.00g                                                          /dev/sda(0)                          
  [m_mimage_0]      t  iwi-aom--- 2.00g                                                          /dev/sdab(0)                         
  [m_mimage_0]      t  iwi-aom--- 2.00g                                                          /dev/sdad(0)                         
  [m_mimage_1]      t  iwi-aom--- 2.00g                                                          /dev/sdaa(0)                         
  [m_mimage_1]      t  iwi-aom--- 2.00g                                                          /dev/sdac(0)                         
  [m_mimage_1]      t  iwi-aom--- 2.00g                                                          /dev/sdae(0)                         
  [m_mlog]          t  mwi-aom--- 4.00m                                         100.00           m_mlog_mimage_0(0),m_mlog_mimage_1(0)
  [m_mlog_mimage_0] t  iwi-aom--- 4.00m                                                          /dev/sdy(0)                          
  [m_mlog_mimage_1] t  iwi-aom--- 4.00m                                                          /dev/sdz(0)
Comment 6 Corey Marthaler 2020-07-06 17:57:52 UTC 
I'm still able to hit this with the latest 7.9 rpms.

3.10.0-1153.el7.x86_64

lvm2-2.02.187-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
lvm2-libs-2.02.187-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
lvm2-cluster-2.02.187-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
lvm2-lockd-2.02.187-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
lvm2-python-boom-1.2-2.el7    BUILT: Mon Jun 29 07:03:49 CDT 2020
cmirror-2.02.187-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
device-mapper-1.02.170-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
device-mapper-libs-1.02.170-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
device-mapper-event-1.02.170-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020
device-mapper-event-libs-1.02.170-6.el7    BUILT: Mon Jun 29 07:02:09 CDT 2020


[root@harding-02 ~]# pvscan
  PV /dev/mapper/mpatha1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathb1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathc1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathd1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathe1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathf1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]
  PV /dev/mapper/mpathg1   VG mirror_sanity     lvm2 [249.96 GiB / 249.96 GiB free]

[root@harding-02 ~]# lvcreate  --type mirror -m 1 --mirrorlog core -R 16 -n convert -L 600M mirror_sanity
  Logical volume "convert" created.

[root@harding-02 ~]# lvconvert --type mirror -m 1 --mirrorlog disk mirror_sanity/convert
  Logical volume mirror_sanity/convert converted.

[root@harding-02 ~]# lvconvert --type mirror -m 1 --mirrorlog mirrored mirror_sanity/convert
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  LV convert_mlog, segment 1 invalid: region size is bigger then LV itself for mirror segment.
  Internal error: LV segments corrupted in convert_mlog.
Comment 7 Heinz Mauelshagen 2020-07-06 23:55:49 UTC 
Corey,

missed the conversion to disk upfront the one to mirrorlog which makes the difference causing the error.
Also, when using regionsize 8m, conversion to mirrorlog's not even being performed from disk at all...
Comment 9 Heinz Mauelshagen 2020-07-09 12:44:50 UTC 
lvm commit ids:

master -> 286a793c12aac1fde17cb9768bacad660e1dfeb3 

stable-2.02 -> 7ebe1bc901c3ed9404f92c7415c7c491593aed47
Comment 15 Corey Marthaler 2020-10-29 18:33:41 UTC 
Doesn't appear to be fixed in the latest 7.9.z rpms. Future fixes should include devel test results.

3.10.0-1160.el7.x86_64

lvm2-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020
lvm2-libs-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020
lvm2-cluster-2.02.187-6.el7_9.2    BUILT: Mon Oct 26 15:21:13 CDT 2020

[root@mckinley-01 ~]# vgs
  VG               #PV #LV #SN Attr   VSize    VFree 
  mirror_sanity      7   0   0 wz--n-   <1.71t <1.71t
  rhel_mckinley-01   1   3   0 wz--n- <556.75g     0 

[root@mckinley-01 ~]# lvcreate  --type mirror -m 1 --mirrorlog core -R 16 -n convert -L 600M mirror_sanity
  Logical volume "convert" created.

[root@mckinley-01 ~]# lvconvert --type mirror -m 1 --mirrorlog disk mirror_sanity/convert
  Logical volume mirror_sanity/convert converted.

[root@mckinley-01 ~]# lvconvert --type mirror -m 1 --mirrorlog mirrored mirror_sanity/convert
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  WARNING: Log type "mirrored" is DEPRECATED and will be removed in the future. Use RAID1 LV or disk log instead.
  device-mapper: reload ioctl on  (253:22) failed: Invalid argument
  Failed to lock logical volume mirror_sanity/convert.

[root@mckinley-01 ~]# echo $?
5

Oct 29 13:27:29 mckinley-01 kernel: device-mapper: dirty region log: invalid region size 32768
Oct 29 13:27:29 mckinley-01 kernel: device-mapper: table: 253:22: mirror: Error creating mirror dirty log
Oct 29 13:27:29 mckinley-01 kernel: device-mapper: ioctl: error adding target to table
Note You need to log in before you can comment on or make changes to this bug.