1249623 – LVM thinp: lvextend fails after thinpool is full

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1249623 - LVM thinp: lvextend fails after thinpool is full

Summary: LVM thinp: lvextend fails after thinpool is full

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Zdenek Kabelac
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1285218
TreeView+	depends on / blocked

Reported:	2015-08-03 12:35 UTC by Bruno Goncalves
Modified:	2023-03-08 07:27 UTC (History)
CC List:	11 users (show)
Fixed In Version:	lvm2-2.02.130-2.el7
Doc Type:	Bug Fix
Doc Text:	This is in-release bug, no doc text required.
Clone Of:
Clones:	1285218 (view as bug list)
Environment:
Last Closed:	2015-11-19 12:47:52 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:2147	0	normal	SHIPPED_LIVE	lvm2 bug fix and enhancement update	2015-11-19 11:11:07 UTC

Description Bruno Goncalves 2015-08-03 12:35:28 UTC

Description of problem:
After creating a thin volume and using all the space on thinpool it was not possible to run lvextend.

Version-Release number of selected component (if applicable):
lvm2-2.02.126-1.el7.x86_64
3.10.0-229.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.# dd if=/dev/zero of=thin_test.img count=0 seek=2048 bs=1M

2.# losetup /dev/loop0 thin_test.img

3.# vgcreate vgtest /dev/loop0

4.# lvcreate -L 200M -T vgtest/mythinpool -V1G -n thin1
  Logical volume "thin1" created.

5.Fill up thin pool by writing on thin1
# dd if=/dev/zero of=/dev/mapper/vgtest-thin1 bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 69.0271 s, 15.6 MB/s

6.Create new LV, as there is no space on pool it SHOULD FAIL
# lvcreate -n thin2 -V 1G -T vgtest/mythinpool
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool vgtest/mythinpool and the size of whole volume group (2.00 GiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Free space in pool vgtest/mythinpool is above threshold, new volumes are not allowed.

7.# lvextend -L+200M vgtest/mythinpool
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Size of logical volume vgtest/mythinpool_tdata changed from 200.00 MiB (50 extents) to 400.00 MiB (100 extents).
  WARNING: Sum of all thin volume sizes (1.00 GiB) exceeds the size of thin pools (200.00 MiB)!
  Thin pool vgtest-mythinpool-tpool (253:5) transaction_id is 2, while expected 1.
  Problem reactivating logical volume vgtest/thin1.
  Releasing activation in critical section.
  libdevmapper exiting with 1 device(s) still suspended.
Actual results:


Expected results:
# lvextend -L+200M vgtest/mythinpool
  Size of logical volume vgtest/mythinpool_tdata changed from 200.00 MiB (50 extents) to 400.00 MiB (100 extents).
  Logical volume mythinpool successfully resized

Comment 2 Bruno Goncalves 2015-08-03 12:37:08 UTC

With lvm2-2.02.115-3.el7.x86_64 (default of RHEL-7.1) it works well.



Trying to create new LV, as there is no space on pool it SHOULD FAIL
# lvcreate -n thin2 -V 1G -T vgtest/mythinpool
  device-mapper: message ioctl on  failed: Invalid argument
  Failed to resume mythinpool.

# dmsetup status
vgtest-thin1: 0 2097152 thin 409600 409599
vgtest-mythinpool-tpool: 0 409600 thin-pool 1 35/1024 3200/3200 - ro discard_passdown queue_if_no_space 
vgtest-mythinpool_tdata: 0 409600 linear 
vgtest-mythinpool_tmeta: 0 8192 linear 
vgtest-mythinpool: 0 409600 linear 
rhel_dell--pe--sc1435--02-home: 0 862363648 linear 
rhel_dell--pe--sc1435--02-swap: 0 8388608 linear 
rhel_dell--pe--sc1435--02-root: 0 104857600 linear 

Going to extend thin pool
# lvextend -L+200M vgtest/mythinpool
  Size of logical volume vgtest/mythinpool_tdata changed from 200.00 MiB (50 extents) to 400.00 MiB (100 extents).
  Logical volume mythinpool successfully resized

Comment 9 Zdenek Kabelac 2015-09-15 09:02:50 UTC

This particular case should be fixed upstream with commit:
https://www.redhat.com/archives/lvm-devel/2015-September/msg00106.html

Comment 11 Corey Marthaler 2015-10-07 22:32:15 UTC

Fixed verified in the latest rpms.

3.10.0-319.el7.x86_64
lvm2-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-libs-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
lvm2-cluster-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-event-libs-1.02.107-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
device-mapper-persistent-data-0.5.5-1.el7    BUILT: Thu Aug 13 09:58:10 CDT 2015
cmirror-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015
sanlock-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
sanlock-lib-3.2.4-1.el7    BUILT: Fri Jun 19 12:48:49 CDT 2015
lvm2-lockd-2.02.130-2.el7    BUILT: Tue Sep 15 07:15:40 CDT 2015


[root@host-116 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  POOL            snapper_thinp twi-aot---   1.00g             0.04   1.56                             POOL_tdata(0) 
  [POOL_tdata]    snapper_thinp Twi-ao----   1.00g                                                     /dev/sdb1(1)  
  [POOL_tmeta]    snapper_thinp ewi-ao----   4.00m                                                     /dev/sdf1(0)  
  display_snap    snapper_thinp Vwi-a-t---   1.00g POOL origin 0.01                                                  
  [lvol0_pmspare] snapper_thinp ewi-------   4.00m                                                     /dev/sdb1(0)  
  origin          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  
  other1          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  
  other2          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  
  other3          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  
  other4          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  
  other5          snapper_thinp Vwi-a-t---   1.00g POOL        0.01                                                  

[root@host-116 ~]# dd if=/dev/zero of=/dev/snapper_thinp/other1 bs=1M count=1100
dd: error writing â/dev/snapper_thinp/other1â: No space left on device
1025+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 65.7462 s, 16.3 MB/s
dd: closing input file â/dev/zeroâ: Bad file descriptor

Oct  7 17:22:23 host-116 lvm[1486]: Thin snapper_thinp-POOL-tpool is now 97% full.
[779880.641370] device-mapper: thin: 253:4: reached low water mark for data device: sending event.
Oct  7 17:22:55 host-116 kernel: device-mapper: thin: 253:4: reached low water mark for data device: sending event.
[779880.650264] device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
Oct  7 17:22:55 host-116 kernel: device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode
Oct  7 17:22:55 host-116 lvm[1486]: Thin snapper_thinp-POOL-tpool is now 100% full.
Oct  7 17:23:55 host-116 kernel: device-mapper: thin: 253:4: switching pool to out-of-data-space (error IO) mode
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262064
Oct  7 17:23:55 host-116 kernel: lost page write due to I/O error on dm-7
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262065
Oct  7 17:23:55 host-116 kernel: lost page write due to I/O error on dm-7
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262066
Oct  7 17:23:55 host-116 kernel: lost page write due to I/O error on dm-7
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262067
Oct  7 17:23:55 host-116 kernel: lost page write due to I/O error on dm-7
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262068
Oct  7 17:23:55 host-116 kernel: lost page write due to I/O error on dm-7
Oct  7 17:23:55 host-116 kernel: Buffer I/O error on device dm-7, logical block 262069

[root@host-116 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize Pool Origin Data%  Meta%  Devices
  POOL            snapper_thinp twi-aot-D- 1.00g             100.00 14.16  POOL_tdata(0)
  [POOL_tdata]    snapper_thinp Twi-ao---- 1.00g                           /dev/sdb1(1)
  [POOL_tmeta]    snapper_thinp ewi-ao---- 4.00m                           /dev/sdf1(0)
  display_snap    snapper_thinp Vwi-a-t--- 1.00g POOL origin 0.01
  [lvol0_pmspare] snapper_thinp ewi------- 4.00m                           /dev/sdb1(0)
  origin          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other1          snapper_thinp Vwi-a-t--- 1.00g POOL        99.97
  other2          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other3          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other4          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other5          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01

[root@host-116 ~]# lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other6
  WARNING: Sum of all thin volume sizes (8.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Free space in pool snapper_thinp/POOL is above threshold, new volumes are not allowed.

[root@host-116 ~]# lvextend -L+500M snapper_thinp/POOL
  WARNING: Sum of all thin volume sizes (7.00 GiB) exceeds the size of thin pools (1.00 GiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Size of logical volume snapper_thinp/POOL_tdata changed from 1.00 GiB (256 extents) to 1.49 GiB (381 extents).
  Logical volume POOL successfully resized.

[root@host-116 ~]# lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other6
  WARNING: Sum of all thin volume sizes (8.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.49 GiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Logical volume "other6" created.

[root@host-116 ~]# lvs -a -o +devices
  LV              VG            Attr       LSize Pool Origin Data%  Meta%  Devices
  POOL            snapper_thinp twi-aot--- 1.49g             67.20  14.26  POOL_tdata(0)
  [POOL_tdata]    snapper_thinp Twi-ao---- 1.49g                           /dev/sdb1(1)
  [POOL_tmeta]    snapper_thinp ewi-ao---- 4.00m                           /dev/sdf1(0)
  display_snap    snapper_thinp Vwi-a-t--- 1.00g POOL origin 0.01
  [lvol0_pmspare] snapper_thinp ewi------- 4.00m                           /dev/sdb1(0)
  origin          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other1          snapper_thinp Vwi-a-t--- 1.00g POOL        99.97
  other2          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other3          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other4          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other5          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01
  other6          snapper_thinp Vwi-a-t--- 1.00g POOL        0.01

Comment 12 errata-xmlrpc 2015-11-19 12:47:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2147.html

Note You need to log in before you can comment on or make changes to this bug.