Bug 696251
| Summary: | apparent "corruption" when attempting to extend mirror using cling tags beyond available extents | |||
|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Corey Marthaler <cmarthal> | |
| Component: | lvm2 | Assignee: | Peter Rajnoha <prajnoha> | |
| Status: | CLOSED ERRATA | QA Contact: | Corey Marthaler <cmarthal> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 6.1 | CC: | agk, dwysocha, heinzm, jbrassow, mbroz, prajnoha, prockai, thornber, zkabelac | |
| Target Milestone: | rc | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | lvm2-2.02.87-1.el6 | Doc Type: | Bug Fix | |
| Doc Text: |
Extending a mirror volume beyond available extents while using "cling by tags" allocation policy was not working correctly before. Normally, such action should end up with an error message that there are insufficient allocatable extents for use. However, this check failed and we could end up with a corrupted volume instead. Since the allocation code has been revised, restructured and made more robust, this change fixed the problematic scenario with extending mirror volumes while using the "cling by tags" policy as well.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 711185 (view as bug list) | Environment: | ||
| Last Closed: | 2011-12-06 16:54:47 UTC | Type: | --- | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 711185 | |||
Can you post the -vvvv log for the lvxtend that fails? first try to reproduce with upstream code I was not able to reproduce the problem with "lvextend --alloc cling", I ended up with "Insufficient free space" message which is what should had also happened here. But on the other hand, I got this problem with inconsistent PV areas with a simple "lvextend" without the alloc specification. I was not able to reproduce at all with current upstream. If can't reproduce with current upstream, then assume it'll be fixed by the 6.2 upstream rebase with no further diagnostic work required. The key to this failure is the extend needs to be larger then the number of tagged extents, but less then the number of extents free in the VG (i.e. this would work if you weren't using tags). In this case the magic number is 36G. [root@taft-01 ~]# lvs -a -o +devices LV VG Attr LSize Log Copy% Devices mirror taft mwi-a- 65.00G mirror_mlog 25.01 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] taft Iwi-ao 65.00G /dev/sdb1(0) [mirror_mimage_0] taft Iwi-ao 65.00G /dev/sdc3(0) [mirror_mimage_1] taft Iwi-ao 65.00G /dev/sdb2(0) [mirror_mimage_1] taft Iwi-ao 65.00G /dev/sdb3(0) [mirror_mlog] taft lwi-ao 4.00M /dev/sdc4(0) [root@taft-01 ~]# pvs -o +pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sda2 VolGroup00 lvm2 a- 68.12G 0 /dev/sdb1 taft lvm2 a- 33.92G 0 A /dev/sdb2 taft lvm2 a- 33.91G 0 B /dev/sdb3 taft lvm2 a- 33.92G 2.83G B /dev/sdb4 taft lvm2 a- 33.91G 33.91G C /dev/sdc1 taft lvm2 a- 33.92G 33.92G C /dev/sdc2 taft lvm2 a- 33.91G 33.91G C /dev/sdc3 taft lvm2 a- 33.92G 2.84G A /dev/sdc4 taft lvm2 a- 33.91G 33.91G A 33.91+2.83=36.74 [root@taft-01 ~]# lvextend --alloc cling_by_tags -L +37G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 102.00 GB Insufficient suitable allocatable extents for logical volume mirror: 132 more required [root@taft-01 ~]# lvextend --alloc cling_by_tags -L +36G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 101.00 GB Inconsistent length: 8487 8683 PV segment pe_alloc_count mismatch: 8487 != 16979 PV segment VG free_count mismatch: 26431 != 17939 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 LV mirror_mimage_1: segment 4 has inconsistent PV area 0 LV mirror_mimage_1: segment 5 has inconsistent PV area 0 LV mirror_mimage_1: segment 6 has inconsistent PV area 0 LV mirror_mimage_1: segment 7 has inconsistent PV area 0 LV mirror_mimage_1: segment 8 has inconsistent PV area 0 LV mirror_mimage_1: segment 9 has inconsistent PV area 0 LV mirror_mimage_1: segment 10 has inconsistent PV area 0 LV mirror_mimage_1: segment 11 has inconsistent PV area 0 LV mirror_mimage_1: segment 12 has inconsistent PV area 0 LV mirror_mimage_1: segment 13 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. Corey, I tried to reproduce this, but unfortunately I didn't manage to... I tried it with lvm2 v83-3 from 6.1 and also a new build lvm2 v86-1 for 6.2. Here's the output: [root@node-a ~]# lvs -a -o+devices LV VG Attr LSize Origin Snap% Move Log Copy% Convert Devices lvol0 vg mwl-a- 132.00m lvol0_mlog 100.00 lvol0_mimage_0(0),lvol0_mimage_1(0) [lvol0_mimage_0] vg iwl-ao 132.00m /dev/sda(0) [lvol0_mimage_0] vg iwl-ao 132.00m /dev/sdc(0) [lvol0_mimage_1] vg iwl-ao 132.00m /dev/sdb(0) [lvol0_mimage_1] vg iwl-ao 132.00m /dev/sdg(0) [lvol0_mlog] vg lwl-ao 4.00m /dev/sdh(0) [root@node-a ~]# pvs -o+pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sda vg lvm2 a-- 124.00m 0 A /dev/sdb vg lvm2 a-- 124.00m 0 B /dev/sdc vg lvm2 a-- 124.00m 116.00m B /dev/sdd vg lvm2 a-- 124.00m 124.00m C /dev/sde vg lvm2 a-- 124.00m 124.00m C /dev/sdf vg lvm2 a-- 124.00m 124.00m C /dev/sdg vg lvm2 a-- 124.00m 116.00m A /dev/sdh vg lvm2 a-- 124.00m 120.00m A [root@node-a ~]# lvextend --alloc cling_by_tags -L +356m vg/lvol0 Extending 2 mirror images. Extending logical volume lvol0 to 488.00 MiB Insufficient suitable allocatable extents for logical volume lvol0: 120 more required [root@node-a ~]# lvextend --alloc cling_by_tags -L +352m vg/lvol0 Extending 2 mirror images. Extending logical volume lvol0 to 484.00 MiB Insufficient suitable allocatable extents for logical volume lvol0: 118 more required [root@node-a ~]# lvextend --alloc cling_by_tags -L +350m vg/lvol0 Rounding up size to full physical extent 352.00 MiB Extending 2 mirror images. Extending logical volume lvol0 to 484.00 MiB Insufficient suitable allocatable extents for logical volume lvol0: 118 more required [root@node-a ~]# lvextend --alloc cling_by_tags -L +346m vg/lvol0 Rounding up size to full physical extent 348.00 MiB Extending 2 mirror images. Extending logical volume lvol0 to 480.00 MiB Insufficient suitable allocatable extents for logical volume lvol0: 116 more required --- (interatively calling lvextend to use all space on A and B tagged PVs) [root@node-a ~]# lvextend --alloc cling_by_tags -L +8m vg/lvol0 Extending 2 mirror images. Extending logical volume lvol0 to 248.00 MiB Logical volume lvol0 successfully resized [root@node-a ~]# pvs PV VG Fmt Attr PSize PFree /dev/sda vg lvm2 a-- 124.00m 0 /dev/sdb vg lvm2 a-- 124.00m 0 /dev/sdc vg lvm2 a-- 124.00m 0 /dev/sdd vg lvm2 a-- 124.00m 124.00m /dev/sde vg lvm2 a-- 124.00m 124.00m /dev/sdf vg lvm2 a-- 124.00m 124.00m /dev/sdg vg lvm2 a-- 124.00m 0 /dev/sdh vg lvm2 a-- 124.00m 120.00m (after using all space a correct error message is on output) [root@node-a ~]# lvextend --alloc cling_by_tags -L +8m vg/lvol0 Extending 2 mirror images. Extending logical volume lvol0 to 256.00 MiB Insufficient suitable allocatable extents for logical volume lvol0: 4 more required I was unable to reproduce this with the latest rpms, however, when I downgraded to what was installed in comment #0, i was again able to reproduce this. # CURRENT RPMS: 2.6.32-188.el6.x86_64 lvm2-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 lvm2-libs-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 lvm2-cluster-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 udev-147-2.37.el6 BUILT: Wed Aug 10 07:48:15 CDT 2011 device-mapper-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-libs-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-event-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-event-libs-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 cmirror-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 lvextend -L 2G --alloc cling_by_tags mirror_sanity/cling_mirror Extending 2 mirror images. Extending logical volume cling_mirror to 2.00 GiB Breaking: preferred_count = 2, ix_offset = 2, devices_needed = 2 Insufficient suitable allocatable extents for logical volume cling_mirror: 48 more required # DOWNGRADED RPMS: 2.6.32-188.el6.x86_64 lvm2-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-libs-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-cluster-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 udev-147-2.37.el6 BUILT: Wed Aug 10 07:48:15 CDT 2011 device-mapper-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 cmirror-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvextend -L 2G --alloc cling_by_tags mirror_sanity/cling_mirror Inconsistent length: 280 488 PV segment pe_alloc_count mismatch: 280 != 304 PV segment VG free_count mismatch: 1165 != 1141 Internal error: PV segments corrupted in mirror_sanity. Extending 2 mirror images. Extending logical volume cling_mirror to 2.00 GiB Fix verified in the latest rpms. [root@grant-01 ~]# lvextend -l 20922 --alloc cling_by_tags mirror_sanity/cling_mirror Extending 2 mirror images. Extending logical volume cling_mirror to 81.73 GiB Breaking: preferred_count = 2, ix_offset = 2, devices_needed = 2 Insufficient suitable allocatable extents for logical volume cling_mirror: 13944 more required 2.6.32-192.el6.x86_64 lvm2-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 lvm2-libs-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 lvm2-cluster-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 udev-147-2.37.el6 BUILT: Wed Aug 10 07:48:15 CDT 2011 device-mapper-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-libs-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-event-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 device-mapper-event-libs-1.02.66-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011 cmirror-2.02.87-1.el6 BUILT: Fri Aug 12 06:11:57 CDT 2011
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.
New Contents:
Extending a mirror volume beyond available extents while using "cling by tags" allocation policy was not working correctly before. Normally, such action should end up with an error message that there are insufficient allocatable extents for use. However, this check failed and we could end up with a corrupted volume instead. Since the allocation code has been revised, restructured and made more robust, this change fixed the problematic scenario with extending mirror volumes while using the "cling by tags" policy as well.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2011-1522.html |
Description of problem: Cling policy: cling_tag_list = [ "@A", "@B" ] ### Extending a sync'ed mirror [root@grant-01 ~]# pvs -o +pv_tags PV VG Fmt Attr PSize PFree PV Tags /dev/sdb1 taft lvm2 a- 51.09g 0 A /dev/sdb2 taft lvm2 a- 51.09g 0 B /dev/sdb3 taft lvm2 a- 51.09g 48.17g B /dev/sdb4 taft lvm2 a- 51.09g 51.09g C /dev/sdc1 taft lvm2 a- 68.11g 68.11g C /dev/sdc2 taft lvm2 a- 68.11g 68.11g C /dev/sdc3 taft lvm2 a- 68.11g 65.20g A /dev/sdc4 taft lvm2 a- 68.11g 68.11g A [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 54.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] iwi-ao 54.00g /dev/sdb1(0) [mirror_mimage_0] iwi-ao 54.00g /dev/sdc3(0) [mirror_mimage_1] iwi-ao 54.00g /dev/sdb2(0) [mirror_mimage_1] iwi-ao 54.00g /dev/sdb3(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +100G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 154.00 GiB Inconsistent length: 1682 13078 PV segment pe_alloc_count mismatch: 1682 != 14950 PV segment VG free_count mismatch: 67875 != 54607 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. ### Extending a non sync'ed mirror [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 50.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] iwi-ao 50.00g /dev/sdb1(0) [mirror_mimage_1] iwi-ao 50.00g /dev/sdb2(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 100.00 GiB Logical volume mirror successfully resized [root@grant-01 ~]# lvs -a -o +devices LV Attr LSize Log Copy% Devices mirror Mwi-a- 100.00g mirror_mlog 50.03 mirror_mimage_0(0),mirror_mimage_1(0) [mirror_mimage_0] Iwi-ao 100.00g /dev/sdb1(0) [mirror_mimage_0] Iwi-ao 100.00g /dev/sdc3(0) [mirror_mimage_1] Iwi-ao 100.00g /dev/sdb2(0) [mirror_mimage_1] Iwi-ao 100.00g /dev/sdb3(0) [mirror_mlog] lwi-ao 4.00m /dev/sdc4(0) [root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror Extending 2 mirror images. Extending logical volume mirror to 150.00 GiB Inconsistent length: 12534 13078 PV segment pe_alloc_count mismatch: 12534 != 24778 PV segment VG free_count mismatch: 58047 != 45803 Internal error: PV segments corrupted in taft. LV mirror_mimage_1: segment 3 has inconsistent PV area 0 LV mirror_mimage_1: segment 4 has inconsistent PV area 0 LV mirror_mimage_1: segment 5 has inconsistent PV area 0 LV mirror_mimage_1: segment 6 has inconsistent PV area 0 LV mirror_mimage_1: segment 7 has inconsistent PV area 0 LV mirror_mimage_1: segment 8 has inconsistent PV area 0 LV mirror_mimage_1: segment 9 has inconsistent PV area 0 LV mirror_mimage_1: segment 10 has inconsistent PV area 0 LV mirror_mimage_1: segment 11 has inconsistent PV area 0 LV mirror_mimage_1: segment 12 has inconsistent PV area 0 LV mirror_mimage_1: segment 13 has inconsistent PV area 0 LV mirror_mimage_1: segment 14 has inconsistent PV area 0 LV mirror_mimage_1: segment 15 has inconsistent PV area 0 LV mirror_mimage_1: segment 16 has inconsistent PV area 0 LV mirror_mimage_1: segment 17 has inconsistent PV area 0 LV mirror_mimage_1: segment 18 has inconsistent PV area 0 LV mirror_mimage_1: segment 19 has inconsistent PV area 0 LV mirror_mimage_1: segment 20 has inconsistent PV area 0 LV mirror_mimage_1: segment 21 has inconsistent PV area 0 LV mirror_mimage_1: segment 22 has inconsistent PV area 0 LV mirror_mimage_1: segment 23 has inconsistent PV area 0 LV mirror_mimage_1: segment 24 has inconsistent PV area 0 Internal error: LV segments corrupted in mirror_mimage_1. Version Release number of selected component (if applicable): 2.6.32-130.el6.x86_64 lvm2-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-libs-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 lvm2-cluster-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 udev-147-2.35.el6 BUILT: Wed Mar 30 07:32:05 CDT 2011 device-mapper-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 device-mapper-event-libs-1.02.62-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 cmirror-2.02.83-3.el6 BUILT: Fri Mar 18 09:31:10 CDT 2011 How reproducible: Everytime