RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 696251 - apparent "corruption" when attempting to extend mirror using cling tags beyond available extents
Summary: apparent "corruption" when attempting to extend mirror using cling tags beyon...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Peter Rajnoha
QA Contact: Corey Marthaler
URL:
Whiteboard:
Depends On:
Blocks: 711185
TreeView+ depends on / blocked
 
Reported: 2011-04-13 17:14 UTC by Corey Marthaler
Modified: 2011-12-06 16:54 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.02.87-1.el6
Doc Type: Bug Fix
Doc Text:
Extending a mirror volume beyond available extents while using "cling by tags" allocation policy was not working correctly before. Normally, such action should end up with an error message that there are insufficient allocatable extents for use. However, this check failed and we could end up with a corrupted volume instead. Since the allocation code has been revised, restructured and made more robust, this change fixed the problematic scenario with extending mirror volumes while using the "cling by tags" policy as well.
Clone Of:
: 711185 (view as bug list)
Environment:
Last Closed: 2011-12-06 16:54:47 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:1522 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2011-12-06 00:50:10 UTC

Description Corey Marthaler 2011-04-13 17:14:47 UTC 
Description of problem:
Cling policy:
     cling_tag_list = [ "@A", "@B" ]

### Extending a sync'ed mirror

[root@grant-01 ~]# pvs -o +pv_tags
  PV         VG    Fmt  Attr PSize  PFree  PV Tags
  /dev/sdb1  taft  lvm2 a-   51.09g     0  A
  /dev/sdb2  taft  lvm2 a-   51.09g     0  B
  /dev/sdb3  taft  lvm2 a-   51.09g 48.17g B
  /dev/sdb4  taft  lvm2 a-   51.09g 51.09g C
  /dev/sdc1  taft  lvm2 a-   68.11g 68.11g C
  /dev/sdc2  taft  lvm2 a-   68.11g 68.11g C
  /dev/sdc3  taft  lvm2 a-   68.11g 65.20g A
  /dev/sdc4  taft  lvm2 a-   68.11g 68.11g A

[root@grant-01 ~]# lvs -a -o +devices
  LV                Attr   LSize  Log         Copy%  Devices
  mirror            Mwi-a- 54.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] iwi-ao 54.00g                    /dev/sdb1(0)
  [mirror_mimage_0] iwi-ao 54.00g                    /dev/sdc3(0)
  [mirror_mimage_1] iwi-ao 54.00g                    /dev/sdb2(0)
  [mirror_mimage_1] iwi-ao 54.00g                    /dev/sdb3(0)
  [mirror_mlog]     lwi-ao  4.00m                    /dev/sdc4(0)

[root@grant-01 ~]# lvextend --alloc cling -L +100G taft/mirror
  Extending 2 mirror images.
  Extending logical volume mirror to 154.00 GiB
  Inconsistent length: 1682 13078
  PV segment pe_alloc_count mismatch: 1682 != 14950
  PV segment VG free_count mismatch: 67875 != 54607
  Internal error: PV segments corrupted in taft.
  LV mirror_mimage_1: segment 3 has inconsistent PV area 0
  Internal error: LV segments corrupted in mirror_mimage_1.


### Extending a non sync'ed mirror

[root@grant-01 ~]# lvs -a -o +devices
  LV                Attr   LSize  Log         Copy%  Devices
  mirror            Mwi-a- 50.00g mirror_mlog 100.00 mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] iwi-ao 50.00g                    /dev/sdb1(0)
  [mirror_mimage_1] iwi-ao 50.00g                    /dev/sdb2(0)
  [mirror_mlog]     lwi-ao  4.00m                    /dev/sdc4(0)

[root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror
  Extending 2 mirror images.
  Extending logical volume mirror to 100.00 GiB
  Logical volume mirror successfully resized

[root@grant-01 ~]# lvs -a -o +devices
  LV                Attr   LSize   Log         Copy%  Devices
  mirror            Mwi-a- 100.00g mirror_mlog  50.03 mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] Iwi-ao 100.00g                    /dev/sdb1(0)
  [mirror_mimage_0] Iwi-ao 100.00g                    /dev/sdc3(0)
  [mirror_mimage_1] Iwi-ao 100.00g                    /dev/sdb2(0)
  [mirror_mimage_1] Iwi-ao 100.00g                    /dev/sdb3(0)
  [mirror_mlog]     lwi-ao   4.00m                    /dev/sdc4(0)

[root@grant-01 ~]# lvextend --alloc cling -L +50G taft/mirror
  Extending 2 mirror images.
  Extending logical volume mirror to 150.00 GiB
  Inconsistent length: 12534 13078
  PV segment pe_alloc_count mismatch: 12534 != 24778
  PV segment VG free_count mismatch: 58047 != 45803
  Internal error: PV segments corrupted in taft.
  LV mirror_mimage_1: segment 3 has inconsistent PV area 0
  LV mirror_mimage_1: segment 4 has inconsistent PV area 0
  LV mirror_mimage_1: segment 5 has inconsistent PV area 0
  LV mirror_mimage_1: segment 6 has inconsistent PV area 0
  LV mirror_mimage_1: segment 7 has inconsistent PV area 0
  LV mirror_mimage_1: segment 8 has inconsistent PV area 0
  LV mirror_mimage_1: segment 9 has inconsistent PV area 0
  LV mirror_mimage_1: segment 10 has inconsistent PV area 0
  LV mirror_mimage_1: segment 11 has inconsistent PV area 0
  LV mirror_mimage_1: segment 12 has inconsistent PV area 0
  LV mirror_mimage_1: segment 13 has inconsistent PV area 0
  LV mirror_mimage_1: segment 14 has inconsistent PV area 0
  LV mirror_mimage_1: segment 15 has inconsistent PV area 0
  LV mirror_mimage_1: segment 16 has inconsistent PV area 0
  LV mirror_mimage_1: segment 17 has inconsistent PV area 0
  LV mirror_mimage_1: segment 18 has inconsistent PV area 0
  LV mirror_mimage_1: segment 19 has inconsistent PV area 0
  LV mirror_mimage_1: segment 20 has inconsistent PV area 0
  LV mirror_mimage_1: segment 21 has inconsistent PV area 0
  LV mirror_mimage_1: segment 22 has inconsistent PV area 0
  LV mirror_mimage_1: segment 23 has inconsistent PV area 0
  LV mirror_mimage_1: segment 24 has inconsistent PV area 0
  Internal error: LV segments corrupted in mirror_mimage_1.

Version Release number of selected component (if applicable):
2.6.32-130.el6.x86_64

lvm2-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-libs-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-cluster-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
udev-147-2.35.el6    BUILT: Wed Mar 30 07:32:05 CDT 2011
device-mapper-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
cmirror-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011


How reproducible:
Everytime
Comment 1 Peter Rajnoha 2011-06-06 13:10:39 UTC 
Can you post the -vvvv log for the lvxtend that fails?
Comment 2 Alasdair Kergon 2011-06-09 14:44:34 UTC 
first try to reproduce with upstream code
Comment 3 Peter Rajnoha 2011-06-10 13:37:05 UTC 
I was not able to reproduce the problem with "lvextend --alloc cling", I ended up with "Insufficient free space" message which is what should had also happened here. But on the other hand, I got this problem with inconsistent PV areas with a simple "lvextend" without the alloc specification. I was not able to reproduce at all with current upstream.
Comment 5 Alasdair Kergon 2011-06-10 13:46:44 UTC 
If can't reproduce with current upstream, then assume it'll be fixed by the 6.2 upstream rebase with no further diagnostic work required.
Comment 6 Corey Marthaler 2011-07-06 19:55:38 UTC 
The key to this failure is the extend needs to be larger then the number of
tagged extents, but less then the number of extents free in the VG (i.e. this
would work if you weren't using tags). In this case the magic number is 36G.

[root@taft-01 ~]# lvs -a -o +devices
 LV                VG   Attr   LSize  Log         Copy%  Devices
 mirror            taft mwi-a- 65.00G mirror_mlog  25.01
mirror_mimage_0(0),mirror_mimage_1(0)
 [mirror_mimage_0] taft Iwi-ao 65.00G                    /dev/sdb1(0)
 [mirror_mimage_0] taft Iwi-ao 65.00G                    /dev/sdc3(0)
 [mirror_mimage_1] taft Iwi-ao 65.00G                    /dev/sdb2(0)
 [mirror_mimage_1] taft Iwi-ao 65.00G                    /dev/sdb3(0)
 [mirror_mlog]     taft lwi-ao  4.00M                    /dev/sdc4(0)

[root@taft-01 ~]# pvs -o +pv_tags
  PV         VG         Fmt  Attr PSize  PFree  PV Tags
  /dev/sda2  VolGroup00 lvm2 a-   68.12G     0
  /dev/sdb1  taft       lvm2 a-   33.92G     0  A
  /dev/sdb2  taft       lvm2 a-   33.91G     0  B
  /dev/sdb3  taft       lvm2 a-   33.92G  2.83G B
  /dev/sdb4  taft       lvm2 a-   33.91G 33.91G C
  /dev/sdc1  taft       lvm2 a-   33.92G 33.92G C
  /dev/sdc2  taft       lvm2 a-   33.91G 33.91G C
  /dev/sdc3  taft       lvm2 a-   33.92G  2.84G A
  /dev/sdc4  taft       lvm2 a-   33.91G 33.91G A

33.91+2.83=36.74

[root@taft-01 ~]# lvextend --alloc cling_by_tags -L +37G taft/mirror
  Extending 2 mirror images.
  Extending logical volume mirror to 102.00 GB
  Insufficient suitable allocatable extents for logical volume mirror: 132 more
required

[root@taft-01 ~]# lvextend --alloc cling_by_tags -L +36G taft/mirror
  Extending 2 mirror images.
  Extending logical volume mirror to 101.00 GB
  Inconsistent length: 8487 8683
  PV segment pe_alloc_count mismatch: 8487 != 16979
  PV segment VG free_count mismatch: 26431 != 17939
  Internal error: PV segments corrupted in taft.
  LV mirror_mimage_1: segment 3 has inconsistent PV area 0
  LV mirror_mimage_1: segment 4 has inconsistent PV area 0
  LV mirror_mimage_1: segment 5 has inconsistent PV area 0
  LV mirror_mimage_1: segment 6 has inconsistent PV area 0
  LV mirror_mimage_1: segment 7 has inconsistent PV area 0
  LV mirror_mimage_1: segment 8 has inconsistent PV area 0
  LV mirror_mimage_1: segment 9 has inconsistent PV area 0
  LV mirror_mimage_1: segment 10 has inconsistent PV area 0
  LV mirror_mimage_1: segment 11 has inconsistent PV area 0
  LV mirror_mimage_1: segment 12 has inconsistent PV area 0
  LV mirror_mimage_1: segment 13 has inconsistent PV area 0
  Internal error: LV segments corrupted in mirror_mimage_1.
Comment 7 Peter Rajnoha 2011-08-11 13:42:12 UTC 
Corey, I tried to reproduce this, but unfortunately I didn't manage to... I tried it with lvm2 v83-3 from 6.1 and also a new build lvm2 v86-1 for 6.2. Here's the output:

[root@node-a ~]# lvs -a -o+devices
  LV               VG       Attr   LSize   Origin Snap%  Move Log        Copy%  Convert Devices                            
  lvol0            vg       mwl-a- 132.00m                    lvol0_mlog 100.00         lvol0_mimage_0(0),lvol0_mimage_1(0)
  [lvol0_mimage_0] vg       iwl-ao 132.00m                                              /dev/sda(0)                        
  [lvol0_mimage_0] vg       iwl-ao 132.00m                                              /dev/sdc(0)                        
  [lvol0_mimage_1] vg       iwl-ao 132.00m                                              /dev/sdb(0)                        
  [lvol0_mimage_1] vg       iwl-ao 132.00m                                              /dev/sdg(0)                        
  [lvol0_mlog]     vg       lwl-ao   4.00m                                              /dev/sdh(0)             

[root@node-a ~]# pvs -o+pv_tags
  PV         VG       Fmt  Attr PSize   PFree   PV Tags
  /dev/sda   vg       lvm2 a--  124.00m      0  A      
  /dev/sdb   vg       lvm2 a--  124.00m      0  B      
  /dev/sdc   vg       lvm2 a--  124.00m 116.00m B      
  /dev/sdd   vg       lvm2 a--  124.00m 124.00m C      
  /dev/sde   vg       lvm2 a--  124.00m 124.00m C      
  /dev/sdf   vg       lvm2 a--  124.00m 124.00m C      
  /dev/sdg   vg       lvm2 a--  124.00m 116.00m A      
  /dev/sdh   vg       lvm2 a--  124.00m 120.00m A      

[root@node-a ~]# lvextend --alloc cling_by_tags -L +356m vg/lvol0
  Extending 2 mirror images.
  Extending logical volume lvol0 to 488.00 MiB
  Insufficient suitable allocatable extents for logical volume lvol0: 120 more required

[root@node-a ~]# lvextend --alloc cling_by_tags -L +352m vg/lvol0
  Extending 2 mirror images.
  Extending logical volume lvol0 to 484.00 MiB
  Insufficient suitable allocatable extents for logical volume lvol0: 118 more required

[root@node-a ~]# lvextend --alloc cling_by_tags -L +350m vg/lvol0
  Rounding up size to full physical extent 352.00 MiB
  Extending 2 mirror images.
  Extending logical volume lvol0 to 484.00 MiB
  Insufficient suitable allocatable extents for logical volume lvol0: 118 more required

[root@node-a ~]# lvextend --alloc cling_by_tags -L +346m vg/lvol0
  Rounding up size to full physical extent 348.00 MiB
  Extending 2 mirror images.
  Extending logical volume lvol0 to 480.00 MiB
  Insufficient suitable allocatable extents for logical volume lvol0: 116 more required

---

(interatively calling lvextend to use all space on A and B tagged PVs)

[root@node-a ~]# lvextend --alloc cling_by_tags -L +8m vg/lvol0
  Extending 2 mirror images.
  Extending logical volume lvol0 to 248.00 MiB
  Logical volume lvol0 successfully resized

[root@node-a ~]# pvs
  PV         VG       Fmt  Attr PSize   PFree  
  /dev/sda   vg       lvm2 a--  124.00m      0 
  /dev/sdb   vg       lvm2 a--  124.00m      0 
  /dev/sdc   vg       lvm2 a--  124.00m      0 
  /dev/sdd   vg       lvm2 a--  124.00m 124.00m
  /dev/sde   vg       lvm2 a--  124.00m 124.00m
  /dev/sdf   vg       lvm2 a--  124.00m 124.00m
  /dev/sdg   vg       lvm2 a--  124.00m      0 
  /dev/sdh   vg       lvm2 a--  124.00m 120.00m

(after using all space a correct error message is on output)

[root@node-a ~]# lvextend --alloc cling_by_tags -L +8m vg/lvol0
  Extending 2 mirror images.
  Extending logical volume lvol0 to 256.00 MiB
  Insufficient suitable allocatable extents for logical volume lvol0: 4 more required
Comment 8 Corey Marthaler 2011-08-17 21:07:26 UTC 
I was unable to reproduce this with the latest rpms, however, when I downgraded to what was installed in comment #0, i was again able to reproduce this.


# CURRENT RPMS:

2.6.32-188.el6.x86_64

lvm2-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
lvm2-libs-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
lvm2-cluster-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
udev-147-2.37.el6    BUILT: Wed Aug 10 07:48:15 CDT 2011
device-mapper-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-libs-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-event-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-event-libs-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
cmirror-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011

lvextend -L 2G --alloc cling_by_tags mirror_sanity/cling_mirror
  Extending 2 mirror images.
  Extending logical volume cling_mirror to 2.00 GiB
  Breaking: preferred_count = 2, ix_offset = 2, devices_needed = 2
  Insufficient suitable allocatable extents for logical volume cling_mirror: 48 more required


# DOWNGRADED RPMS:

2.6.32-188.el6.x86_64
lvm2-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-libs-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-cluster-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
udev-147-2.37.el6    BUILT: Wed Aug 10 07:48:15 CDT 2011
device-mapper-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
cmirror-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011

lvextend -L 2G --alloc cling_by_tags mirror_sanity/cling_mirror
  Inconsistent length: 280 488
  PV segment pe_alloc_count mismatch: 280 != 304
  PV segment VG free_count mismatch: 1165 != 1141
  Internal error: PV segments corrupted in mirror_sanity.
  Extending 2 mirror images.
  Extending logical volume cling_mirror to 2.00 GiB
Comment 11 Corey Marthaler 2011-09-06 21:05:44 UTC 
Fix verified in the latest rpms.

[root@grant-01 ~]# lvextend -l 20922 --alloc cling_by_tags mirror_sanity/cling_mirror
  Extending 2 mirror images.
  Extending logical volume cling_mirror to 81.73 GiB
  Breaking: preferred_count = 2, ix_offset = 2, devices_needed = 2
  Insufficient suitable allocatable extents for logical volume cling_mirror: 13944 more required

2.6.32-192.el6.x86_64

lvm2-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
lvm2-libs-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
lvm2-cluster-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
udev-147-2.37.el6    BUILT: Wed Aug 10 07:48:15 CDT 2011
device-mapper-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-libs-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-event-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
device-mapper-event-libs-1.02.66-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
cmirror-2.02.87-1.el6    BUILT: Fri Aug 12 06:11:57 CDT 2011
Comment 12 Peter Rajnoha 2011-10-27 11:57:01 UTC 
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Extending a mirror volume beyond available extents while using "cling by tags" allocation policy was not working correctly before. Normally, such action should end up with an error message that there are insufficient allocatable extents for use. However, this check failed and we could end up with a corrupted volume instead. Since the allocation code has been revised, restructured and made more robust, this change fixed the problematic scenario with extending mirror volumes while using the "cling by tags" policy as well.
Comment 13 errata-xmlrpc 2011-12-06 16:54:47 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1522.html
Note You need to log in before you can comment on or make changes to this bug.