+++ This bug was initially created as a clone of Bug #696251 +++ Description of problem: Cling policy: cling_tag_list = [ "@A", "@B" ] ### Extending a sync'ed mirror [root@grant-01 ~]# pvs -o +pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sdb1 taft lvm2 a- 51.09g 0 A /dev/sdb2 taft lvm2 a- 51.09g 0 B /dev/sdb3 taft lvm2 a- 51.09g 48.17g B /dev/sdb4 taft lvm2 a- 51.09g 51.09g C /dev/sdc1 taft lvm2 a- 68.11g 68.11g C /dev/sdc2 taft lvm2 a- 68.11g 68.11g C /dev/sdc3 taft lvm2 a- 68.11g 65.20g A /dev/sdc4 taft lvm2 a- 68.11g 68.11g A [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 54.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] iwi-ao 54.00g /dev/sdb1(0) [mirror_mimage_0] iwi-ao 54.00g /dev/sdc3(0) [mirror_mimage_1] iwi-ao 54.00g /dev/sdb2(0) [mirror_mimage_1] iwi-ao 54.00g /dev/sdb3(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +100G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 154.00 GiB Inconsistent length: 1682 13078 PV segment pe_alloc_count mismatch: 1682 != 14950 PV segment VG free_count mismatch: 67875 != 54607 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. ### Extending a non sync'ed mirror [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 50.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] iwi-ao 50.00g /dev/sdb1(0) [mirror_mimage_1] iwi-ao 50.00g /dev/sdb2(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 100.00 GiB Logical volume mirror successfully resized [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 100.00g mirror_mlog 50.03 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] Iwi-ao 100.00g /dev/sdb1(0) [mirror_mimage_0] Iwi-ao 100.00g /dev/sdc3(0) [mirror_mimage_1] Iwi-ao 100.00g /dev/sdb2(0) [mirror_mimage_1] Iwi-ao 100.00g /dev/sdb3(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 150.00 GiB Inconsistent length: 12534 13078 PV segment pe_alloc_count mismatch: 12534 != 24778 PV segment VG free_count mismatch: 58047 != 45803 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 LV mirror_mimage_1: segment 4 has inconsistent PV area 0 LV mirror_mimage_1: segment 5 has inconsistent PV area 0 LV mirror_mimage_1: segment 6 has inconsistent PV area 0 LV mirror_mimage_1: segment 7 has inconsistent PV area 0 LV mirror_mimage_1: segment 8 has inconsistent PV area 0 LV mirror_mimage_1: segment 9 has inconsistent PV area 0 LV mirror_mimage_1: segment 10 has inconsistent PV area 0 LV mirror_mimage_1: segment 11 has inconsistent PV area 0 LV mirror_mimage_1: segment 12 has inconsistent PV area 0 LV mirror_mimage_1: segment 13 has inconsistent PV area 0 LV mirror_mimage_1: segment 14 has inconsistent PV area 0 LV mirror_mimage_1: segment 15 has inconsistent PV area 0 LV mirror_mimage_1: segment 16 has inconsistent PV area 0 LV mirror_mimage_1: segment 17 has inconsistent PV area 0 LV mirror_mimage_1: segment 18 has inconsistent PV area 0 LV mirror_mimage_1: segment 19 has inconsistent PV area 0 LV mirror_mimage_1: segment 20 has inconsistent PV area 0 LV mirror_mimage_1: segment 21 has inconsistent PV area 0 LV mirror_mimage_1: segment 22 has inconsistent PV area 0 LV mirror_mimage_1: segment 23 has inconsistent PV area 0 LV mirror_mimage_1: segment 24 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. Version Release number of selected component (if applicable): 2.6.32-130.el6.x86_64 lvm2-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-libs-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-cluster-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 udev-147-2.35.el6 BUILT: Wed Mar 30 07:32:05 CDT 2011 device-mapper-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 cmirror-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 How reproducible: Everytime --- Additional comment from prajnoha on 2011-06-06 09:10:39 EDT --- Can you post the -vvvv log for the lvxtend that fails?
Here's the RHEL5.7 reproduction of this issue: [root@grant-03 ~]# lvs -a -o +devices LV VG Attr LSize Log Copy% Devices mirror grant mwi-a- 90.00G mirror_mlog 2.09 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] grant Iwi-ao 90.00G /dev/sdb1(0) [mirror_mimage_0] grant Iwi-ao 90.00G /dev/sdc3(0) [mirror_mimage_1] grant Iwi-ao 90.00G /dev/sdb2(0) [mirror_mimage_1] grant Iwi-ao 90.00G /dev/sdb3(0) [mirror_mlog] grant lwi-ao 4.00M /dev/sdc4(0) [root@grant-03 ~]# lvextend --alloc cling -L +50G grant/mirror Extending 2 mirror images. Extending logical volume mirror to 140.00 GB Inconsistent length: 10298 13078 PV segment pe_alloc_count mismatch: 10298 != 19982 PV segment VG free_count mismatch: 62843 != 53159 Internal error: PV segments corrupted in grant. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 LV mirror_mimage_1: segment 4 has inconsistent PV area 0 LV mirror_mimage_1: segment 5 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. Version Release number of selected component (if applicable): 2.6.18-261.el5 lvm2-2.02.84-3.el5 BUILT: Wed Apr 27 03:42:24 CDT 2011 lvm2-cluster-2.02.84-3.el5 BUILT: Wed Apr 27 03:42:43 CDT 2011 device-mapper-1.02.63-3.el5 BUILT: Thu May 19 08:09:22 CDT 2011 device-mapper-event-1.02.63-3.el5 BUILT: Thu May 19 08:09:22 CDT 2011 cmirror-1.1.39-10.el5 BUILT: Wed Sep 8 16:32:05 CDT 2010 kmod-cmirror-0.1.22-3.el5 BUILT: Tue Dec 22 13:39:47 CST 2009
Created attachment 503306 [details] lvextend -vvvv from grant-01
The key to this failure
The key to this failure is the extend needs to be larger then the number of tagged extents, but less then the number of extents free in the VG (i.e. this would work if you weren't using tags). In this case the magic number is 36G. [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Log Copy% Devices mirror taft mwi-a- 65.00G mirror_mlog 25.01 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] taft Iwi-ao 65.00G /dev/sdb1(0) [mirror_mimage_0] taft Iwi-ao 65.00G /dev/sdc3(0) [mirror_mimage_1] taft Iwi-ao 65.00G /dev/sdb2(0) [mirror_mimage_1] taft Iwi-ao 65.00G /dev/sdb3(0) [mirror_mlog] taft lwi-ao 4.00M /dev/sdc4(0) [root@taft-01 ~]# pvs -o +pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sda2 VolGroup00 lvm2 a- 68.12G 0 /dev/sdb1 taft lvm2 a- 33.92G 0 A /dev/sdb2 taft lvm2 a- 33.91G 0 B /dev/sdb3 taft lvm2 a- 33.92G 2.83G B /dev/sdb4 taft lvm2 a- 33.91G 33.91G C /dev/sdc1 taft lvm2 a- 33.92G 33.92G C /dev/sdc2 taft lvm2 a- 33.91G 33.91G C /dev/sdc3 taft lvm2 a- 33.92G 2.84G A /dev/sdc4 taft lvm2 a- 33.91G 33.91G A 33.91+2.83=36.74 [root@taft-01 ~]# lvextend --alloc cling_by_tags -L +37G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 102.00 GB Insufficient suitable allocatable extents for logical volume mirror: 132 more required [root@taft-01 ~]# lvextend --alloc cling_by_tags -L +36G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 101.00 GB Inconsistent length: 8487 8683 PV segment pe_alloc_count mismatch: 8487 != 16979 PV segment VG free_count mismatch: 26431 != 17939 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 LV mirror_mimage_1: segment 4 has inconsistent PV area 0 LV mirror_mimage_1: segment 5 has inconsistent PV area 0 LV mirror_mimage_1: segment 6 has inconsistent PV area 0 LV mirror_mimage_1: segment 7 has inconsistent PV area 0 LV mirror_mimage_1: segment 8 has inconsistent PV area 0 LV mirror_mimage_1: segment 9 has inconsistent PV area 0 LV mirror_mimage_1: segment 10 has inconsistent PV area 0 LV mirror_mimage_1: segment 11 has inconsistent PV area 0 LV mirror_mimage_1: segment 12 has inconsistent PV area 0 LV mirror_mimage_1: segment 13 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1.
It appears that another key to this issue is to have mismatched sized PVs where the primary and secondary legs are placed. Here the primary PV is 4.92G and the secondary PV is 4.91. [root@taft-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices cling_mirror mwi-a- 3.00G cling_mirror_mlog 13.02 cling_mirror_mimage_0(0),cling_mirror_mimage_1(0) [cling_mirror_mimage_0] Iwi-ao 3.00G /dev/sdh2(0) [cling_mirror_mimage_1] Iwi-ao 3.00G /dev/sdd1(0) [cling_mirror_mlog] lwi-ao 4.00M /dev/sde1(50) [root@taft-01 ~]# pvs -o +pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sdd1 mirror_sanity lvm2 a- 4.91G 1.91G B /dev/sdd2 mirror_sanity lvm2 a- 4.92G 4.92G C /dev/sde1 mirror_sanity lvm2 a- 4.91G 4.91G A /dev/sdh2 mirror_sanity lvm2 a- 4.92G 1.92G A [root@taft-01 ~]# lvextend --alloc cling_by_tags -L +2G mirror_sanity/cling_mirror Extending 2 mirror images. Extending logical volume cling_mirror to 5.00 GB Inconsistent length: 791 1257 PV segment pe_alloc_count mismatch: 791 != 814 PV segment VG free_count mismatch: 2960 != 2937 Internal error: PV segments corrupted in mirror_sanity.
Fixed in lvm2-2.02.88-1.el5.
Fix verified in the latest rpms. 2.6.18-274.el5 lvm2-2.02.88-4.el5 BUILT: Wed Nov 16 09:40:55 CST 2011 lvm2-cluster-2.02.88-4.el5 BUILT: Wed Nov 16 09:46:51 CST 2011 device-mapper-1.02.67-2.el5 BUILT: Mon Oct 17 08:31:56 CDT 2011 device-mapper-event-1.02.67-2.el5 BUILT: Mon Oct 17 08:31:56 CDT 2011 cmirror-1.1.39-10.el5 BUILT: Wed Sep 8 16:32:05 CDT 2010 kmod-cmirror-0.1.22-3.el5 BUILT: Tue Dec 22 13:39:47 CST 2009 [root@grant-01 tmp]# pvs -a -o +pv_tags /dev/sdb1 mirror_sanity lvm2 a-- 40.87G 40.87G /dev/sdb2 mirror_sanity lvm2 a-- 40.87G 40.87G /dev/sdb3 mirror_sanity lvm2 a-- 40.87G 0 A /dev/sdb5 mirror_sanity lvm2 a-- 40.88G 8.00M B /dev/sdb6 mirror_sanity lvm2 a-- 40.87G 40.86G /dev/sdc1 mirror_sanity lvm2 a-- 54.49G 54.49G A /dev/sdc2 mirror_sanity lvm2 a-- 54.49G 54.49G /dev/sdc3 mirror_sanity lvm2 a-- 54.48G 54.48G /dev/sdc5 mirror_sanity lvm2 a-- 54.49G 54.49G /dev/sdc6 mirror_sanity lvm2 a-- 54.48G 54.48G [root@grant-01 tmp]# lvs -a -o +devices LV VG Attr LSize Log Copy% Devices cling_mirror mirror_sanity mwi-a- 40.87G cling_mirror_mlog 5.46 cling_mirror_mimage_0(0),cling_mirror_mimage_1(0) [cling_mirror_mimage_0] mirror_sanity Iwi-ao 40.87G /dev/sdb3(0) [cling_mirror_mimage_1] mirror_sanity Iwi-ao 40.87G /dev/sdb5(0) [cling_mirror_mlog] mirror_sanity lwi-ao 4.00M /dev/sdb6(50) [root@grant-01 tmp]# lvextend -l 20924 --alloc cling_by_tags mirror_sanity/cling_mirror Extending 2 mirror images. Extending logical volume cling_mirror to 81.73 GB Insufficient suitable allocatable extents for logical volume cling_mirror: 20920 more required
Technical note added. If any revisions are required, please edit the "Technical Notes" field accordingly. All revisions will be proofread by the Engineering Content Services team. New Contents: Extending a mirror volume beyond available extents while using "cling by tags" allocation policy was not working correctly before. Normally, such action should end up with an error message that there are insufficient allocatable extents for use. However, this check failed and we could end up with a corrupted volume instead. Since the allocation code has been revised, restructured and made more robust, this change fixed the problematic scenario with extending mirror volumes while using the "cling by tags" policy as well.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2012-0161.html