Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 651102

Summary:	lvm commands block while online resize is taking place
Product:	Red Hat Enterprise Linux 7	Reporter:	Corey Marthaler <cmarthal>
Component:	lvm2	Assignee:	LVM and device-mapper development team <lvm-team>
lvm2 sub component:	Default / Unclassified	QA Contact:	cluster-qe <cluster-qe>
Status:	CLOSED DEFERRED	Docs Contact:
Severity:	medium
Priority:	medium	CC:	agk, cbuissar, cmarthal, coughlan, dwysocha, heinzm, jbrassow, joe.thornber, msnitzer, okozina, prajnoha, prockai, zkabelac
Version:	7.3	Keywords:	Triaged
Target Milestone:	rc
Target Release:	7.3
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-01-18 23:29:37 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	756082, 1075802, 1159926, 1172231

Description Corey Marthaler 2010-11-08 20:21:43 UTC

Description of problem:
This is related to the fix for bug 641461. When doing the online resize, any lvm cmd to monitor the resize block. However if you use dmsetup, you can tell that progress is being made.

[root@hayes-02 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/mirror_sanity-open_fsadm_resize
                      4.0G  136M  3.7G   4% /mnt/open_fsadm_resize

[root@hayes-02 ~]# mount
/dev/mapper/mirror_sanity-open_fsadm_resize on /mnt/open_fsadm_resize type ext4 (rw)


Attempt to resize both the mirror and the fs with lvextend -r option on hayes-01 while still mounted
hayes-01: lvextend -L +10G -r /dev/mirror_sanity/open_fsadm_resize
resize2fs 1.41.12 (17-May-2010)


[root@hayes-01 ~]# lvs -a -o +devices
[HANG]

[root@hayes-02 ~]# dmsetup status
mirror_sanity-open_fsadm_resize: 0 29360128 mirror 2 253:4 253:5 13166/28672 1 AA 3 clustered-disk 253:3 A

[root@hayes-02 ~]# dmsetup status
mirror_sanity-open_fsadm_resize: 0 29360128 mirror 2 253:4 253:5 13177/28672 1 AA 3 clustered-disk 253:3 A

[root@hayes-02 ~]# dmsetup status
mirror_sanity-open_fsadm_resize: 0 29360128 mirror 2 253:4 253:5 14854/28672 1 AA 3 clustered-disk 253:3 A

Version-Release number of selected component (if applicable):
2.6.32-71.el6.x86_64

lvm2-2.02.72-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
lvm2-libs-2.02.72-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
lvm2-cluster-2.02.72-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
udev-147-2.29.el6    BUILT: Tue Aug 31 16:44:10 CDT 2010
device-mapper-1.02.53-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
device-mapper-libs-1.02.53-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
device-mapper-event-1.02.53-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
device-mapper-event-libs-1.02.53-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010
cmirror-2.02.72-8.el6_0.2    BUILT: Mon Nov  1 11:29:36 CDT 2010


How reproducible:
Everytime

Comment 2 Zdenek Kabelac 2010-11-09 13:23:59 UTC

Resize takes exclusive write lock - so no one else can change it so it can not be dropped for current LVM implementation.

lvs has an option  '--ignorelockingfailure'  (man lvm) to give some results.

Unsure how to proceed here - 

We would probably need few different lock types to make a difference between 

"write - wait for result before read" 
  and

"do not modify, but you can read for now" - something like exclusive READ lock which only this application could switch to write - when all readers are gone.
So we could temporarily drop exclusive write access - and allow parallel read operation to happen - and switch back when needed to WRITE

Comment 3 Zdenek Kabelac 2010-11-09 13:51:48 UTC

As pointed by Alasdair - another way to go is to use  'pvmove' like way of marking LV as metadata locking flags and extend lvresize operation with '--abort' flag.

Comment 4 Alasdair Kergon 2010-11-23 02:48:13 UTC

Slow operations have to be broken up - VG locks are currently designed only to be held for fairly short periods.

Comment 5 Jonathan Earl Brassow 2010-11-30 16:52:56 UTC

Isn't the operation already done?  The extension was made and the metadata written.  Are there any more changes to be made?  As far as I know, it is holding the lock just to tell the user when the rest of the mirror has been synced - it doesn't need a write lock for that.

Does 'lvconvert -m +1' have this same problem?  Isn't that a similar issue?

Comment 6 Corey Marthaler 2011-01-10 22:01:26 UTC

'lvconvert -m +1' hangs other lvm operations until the operation is "done". 

However once the mirror is syncing (but before the 'lvconvert -m +1' operation is finished, all other lvm operations succeed.

[root@grant-01 ~]# lvconvert -m +1 /dev/grant/lv
[all other lvm operations "hang" until the sync starts below]
  grant/lv: Converted: 0.0%
  grant/lv: Converted: 0.1%
  grant/lv: Converted: 0.1%
  grant/lv: Converted: 0.1%
  grant/lv: Converted: 0.1%
  grant/lv: Converted: 0.1%
  grant/lv: Converted: 0.2%
  grant/lv: Converted: 0.2%
  grant/lv: Converted: 0.2%
  grant/lv: Converted: 0.3%
  grant/lv: Converted: 0.3%
  grant/lv: Converted: 0.3%
  grant/lv: Converted: 0.3%
  grant/lv: Converted: 0.4%
  grant/lv: Converted: 0.4%

Comment 7 Tom Coughlan 2011-03-18 15:30:11 UTC

This will not be ready for 6.1. Moving to 6.2.

Comment 9 Corey Marthaler 2011-06-03 18:55:44 UTC

Adding QA ack for 6.2.

Comment 19 Jonathan Earl Brassow 2014-11-21 23:49:21 UTC

Command seems to complete quite quickly now:

[root@bp-03 ~]# lvextend -L +10G -r /dev/vg/lv
  Extending 2 mirror images.
  Size of logical volume vg/lv changed from 4.00 GiB (1024 extents) to 14.00 GiB (3584 extents).
  Logical volume lv successfully resized
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/vg-lv is mounted on /mnt; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/vg-lv is now 3670016 blocks long.

[root@bp-03 ~]# lvs
  LV   VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log     Cpy%Sync Convert
  home rhel_bp-03 -wi-ao---- 407.33g
  root rhel_bp-03 -wi-ao----  50.00g
  swap rhel_bp-03 -wi-ao----   7.88g
  lv   vg         mwi-aom---  14.00g                                lv_mlog 32.14


Could you retest?  (This was performed on rhel7.1 though.)

Comment 21 Corey Marthaler 2014-12-02 19:57:12 UTC

This is no longer reproducable in rhel7.1, however it is still reproducable in rhel6.6.

[root@hayes-01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/mirror_sanity-open_fsadm_resize
                      4.8G   10M  4.6G   1% /mnt/other1

[root@hayes-01 ~]# lvextend -L +175G -r /dev/mirror_sanity/open_fsadm_resize
  Extending 2 mirror images.
  Size of logical volume mirror_sanity/open_fsadm_resize changed from 5.00 GiB (1280 extents) to 180.00 GiB (46080 extents).
  Logical volume open_fsadm_resize successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/mirror_sanity-open_fsadm_resize is mounted on /mnt/other1; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 12
Performing an on-line resize of /dev/mapper/mirror_sanity-open_fsadm_resize to 47185920 (4k) blocks.
The filesystem on /dev/mapper/mirror_sanity-open_fsadm_resize is now 47185920 blocks long.

[WHILE ABOVE IS TAKING PLACE]

[root@hayes-01 ~]# lvs -a -o +devices
^C  Interrupted...
  Giving up waiting for lock.
  /var/lock/lvm/V_mirror_sanity: flock failed: Interrupted system call
  Can't get lock for mirror_sanity
  Skipping volume group mirror_sanity
  Interrupted...
  Giving up waiting for lock.
  Interrupted...
  Giving up waiting for lock.
  Interrupted...
  Internal error: Volume Group vg_hayes01 was not unlocked
  Device '/dev/sda2' has been left open (0 remaining references).
  Internal error: 1 device(s) were left open and have been closed.
  You have a memory leak (not released memory pool):
   [0x7ffe71432e80] format_instance
   [0x7ffe718849a0] read_vg
   [0x7ffe71433b50] read_vg
  Internal error: Unreleased memory pool(s) found.

[root@hayes-01 ~]# time lvs -a -o +devices
real    0m45.039s
user    0m0.364s
sys     0m0.045s


2.6.32-502.el6.x86_64

lvm2-2.02.111-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
lvm2-libs-2.02.111-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
lvm2-cluster-2.02.111-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
udev-147-2.57.el6    BUILT: Thu Jul 24 08:48:47 CDT 2014
device-mapper-1.02.90-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
device-mapper-libs-1.02.90-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
device-mapper-event-1.02.90-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
device-mapper-event-libs-1.02.90-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014
device-mapper-persistent-data-0.3.2-1.el6    BUILT: Fri Apr  4 08:43:06 CDT 2014
cmirror-2.02.111-2.el6    BUILT: Mon Sep  1 06:46:43 CDT 2014

Comment 22 Jonathan Earl Brassow 2014-12-16 21:58:31 UTC

When we rebase for rhel6, we'll have to try to reproduce this again.  I believe it will likely be fixed.

Comment 24 Jonathan Earl Brassow 2015-03-03 20:40:19 UTC

The block occurs while waiting for the file system to resize when using the '-r' option.  This happens when the file system is xfs or ext4 and when the segment type is raid1 or mirror.

The fix will require some code restructuring that is not appropriate for rhel6.7 release.  The change is more suited to rhel7 and may make its way into rhel6 via a backport if necessary.

In the meantime, the workaround is to grow the logical volume and the file system separately (forego the use of '-r').

Comment 25 Corey Marthaler 2015-05-22 20:31:10 UTC

This affects non raid and mirror types as well in the latest 6.7

# Online mirror resize
[root@mckinley-01 ~]# time  lvs -a -o +devices
  LV                           Attr       LSize   Log                    Cpy%Sync Devices
  open_fsadm_resize            Mwi-aom--- 179.00g open_fsadm_resize_mlog 100.00   open_fsadm_resize_mimage_0(0),open_fsadm_resize_mimage_1(0)
  [open_fsadm_resize_mimage_0] iwi-aom--- 179.00g                                 /dev/mapper/mpathgp2(0)
  [open_fsadm_resize_mimage_0] iwi-aom--- 179.00g                                 /dev/mapper/mpathcp2(0)
  [open_fsadm_resize_mimage_1] iwi-aom--- 179.00g                                 /dev/mapper/mpathgp1(0)
  [open_fsadm_resize_mimage_1] iwi-aom--- 179.00g                                 /dev/mapper/mpathcp1(0)
  [open_fsadm_resize_mlog]     lwi-aom---   4.00m                                 /dev/mapper/mpathbp1(0)

real    1m13.254s
user    0m0.193s
sys     0m0.046s


# Online linear resize
[root@mckinley-01 ~]# time  lvs -a -o +devices
  LV                Attr       LSize   Log Cpy%Sync Devices                
  open_fsadm_resize -wi-ao---- 179.00g              /dev/mapper/mpathgp2(0)
  open_fsadm_resize -wi-ao---- 179.00g              /dev/mapper/mpathgp1(0)

real    0m42.267s
user    0m0.187s
sys     0m0.028s



2.6.32-562.el6.x86_64
lvm2-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
lvm2-libs-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
lvm2-cluster-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
udev-147-2.62.el6    BUILT: Thu Apr 23 05:44:37 CDT 2015
device-mapper-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-libs-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-event-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-event-libs-1.02.95-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015
device-mapper-persistent-data-0.3.2-1.el6    BUILT: Fri Apr  4 08:43:06 CDT 2014
cmirror-2.02.118-2.el6    BUILT: Wed Apr 15 06:34:08 CDT 2015

Comment 26 Alasdair Kergon 2015-07-10 11:44:23 UTC

A candidate now for lvmpolld?

Comment 28 Zdenek Kabelac 2015-10-15 09:58:36 UTC

If we target for lvmpolld - let's move this bug it 7.X.
It's quite late for RHEL6

Comment 32 Jonathan Earl Brassow 2016-01-18 23:29:37 UTC

(In reply to Zdenek Kabelac from comment #28)
> If we target for lvmpolld - let's move this bug it 7.X.
> It's quite late for RHEL6

Yes, but in comment 21, reporter says it is not reproducible in rhel7...  since we don't intend to fix this in RHEL6 and it is already fixed in rhel7, I am closing this bug "DEFERRED".  (Perhaps now that this bug is associated with RHEL7, it would be better as "CURRENTRELEASE"?)