Bug 1459646 - Pool space leak: Shrinking snapshotted volumes retains block mappings after deleting all snapshots
Pool space leak: Shrinking snapshotted volumes retains block mappings after d...
Status: NEW
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2 (Show other bugs)
7.3
All Linux
unspecified Severity medium
: rc
: ---
Assigned To: LVM and device-mapper development team
cluster-qe@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-07 13:10 EDT by bugzilla
Modified: 2017-08-17 14:31 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description bugzilla 2017-06-07 13:10:48 EDT
Description of problem:

If you shrink a thin logical volume that has been snapshotted, and then delete the snapshot, the space that should be freed by the snapshot deletion is not unmapped; it is held by the original device that was shrunken and lvs gives the following warning:
  WARNING: Thin volume data/usage_test maps 1.00 GiB while the size is only 512.00 MiB.

This affects CentOS 6 systems also, but instead of giving the warning it shows thin LV usage above 100%.


Version-Release number of selected component (if applicable):

Linux hvtest1.ewheeler.net 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lvm2-2.02.166-1.el7_3.3.x86_64


How reproducible:

Very


Steps to Reproduce:

1. Create thin logical volume

[root@hvtest1 ~]# lvcreate -T data/pool0 -V1g -n usage_test
  Using default stripesize 64.00 KiB.
  Logical volume "usage_test" created.

2. Fully allocate the volume

[root@hvtest1 ~]# dd if=/dev/zero bs=1M of=/dev/data/usage_test
dd: error writing ‘/dev/data/usage_test’: No space left on device

1025+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 18.6354 s, 57.6 MB/s

[root@hvtest1 ~]# lvs data/usage_test
  LV         VG   Attr       LSize Pool  Origin Data%  Meta%  Move Log Cpy%Sync Convert
  usage_test data Vwi-aotz-- 1.00g pool0        100.00                                 

3. Create a thin snapshot of that volume

[root@hvtest1 ~]# lvcreate -s -n usage_test_snap data/usage_test
  Using default stripesize 64.00 KiB.
  Logical volume "usage_test_snap" created.

[root@hvtest1 ~]# lvs |grep usage_test
  usage_test                                                                data       Vwi-a-tz--   1.00g pool0                                                    100.00                                 
  usage_test_snap                                                           data       Vwi---tz-k   1.00g pool0 usage_test                                                                                

4. Shrink the original volume

[root@hvtest1 ~]# lvresize -L512m data/usage_test
  WARNING: Reducing active logical volume to 512.00 MiB.
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce data/usage_test? [y/n]: y
  Size of logical volume data/usage_test changed from 1.00 GiB (256 extents) to 512.00 MiB (128 extents).
  Logical volume data/usage_test successfully resized.

[root@hvtest1 ~]# lvchange -K -ay data/usage_test_snap

[root@hvtest1 ~]# lvs |grep usage_test
  WARNING: Thin volume data/usage_test maps 1.00 GiB while the size is only 512.00 MiB.
  usage_test                                                                data       Vwi-a-tz-- 512.00m pool0                                                    100.00                                 
  usage_test_snap                                                           data       Vwi-a-tz-k   1.00g pool0 usage_test                                         100.00                                 

5. Remove the snapshot

[root@hvtest1 ~]# lvremove data/usage_test_snap
Do you really want to remove active logical volume data/usage_test_snap? [y/n]: y
  Logical volume "usage_test_snap" successfully removed

6. Note the warning. Also, if you were to inspect thin_dump of the tmeta, you would notice allocations beyond the volume size. In CentOS 6, the 100.00 would show 200.00 instead of giving the warning.

[root@hvtest1 ~]# lvs |grep usage_test
  WARNING: Thin volume data/usage_test maps 1.00 GiB while the size is only 512.00 MiB.
  usage_test                                                                data       Vwi-a-tz-- 512.00m pool0          100.00                 


Actual results:

Pool space is lost and not reclaimed.


Expected results:

Pool space should be freed when the snapshot is deleted.


Additional info:
Comment 2 bugzilla 2017-06-07 13:15:34 EDT
Note that deleting the origin frees 1gb (90g * (.1219 - .1108) == .999g) as expected given the nature of the bug:

[root@hvtest1 ~]# lvs data/pool0
  LV    VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  pool0 data twi-aotz-- 90.00g             12.19  5.17                            
[root@hvtest1 ~]# lvremove data/usage_test
Do you really want to remove active logical volume data/usage_test? [y/n]: y
  Logical volume "usage_test" successfully removed
[root@hvtest1 ~]# lvs data/pool0
  LV    VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  pool0 data twi-aotz-- 90.00g             11.08  5.12
Comment 3 Zdenek Kabelac 2017-06-08 04:35:20 EDT
This is not a new bug  - this is basically yet-to-be-resolved issue where lvm2 needs to figure some nice way how to do in configurable manner and without leaks and to long lock holdings.

For now - whenever user 'lvreduce'  size of thinLV  - the reduced chunk is NOT 'trimmed/discard' by lvm2 (so user may 'revert'  lvreduce step by vgcfgrestore in case he did a mistake - as lvm2 normally tries to support 1-command-back safe recovery)

So the current 'workaround' for a user is to 'discard'  this reduced chunk in-front by issuing 'blkdiscard' command.

Next problem on lvm2 side is - it cannot be done atomically so we need to basically put in internal mechanism to 'queue' some work into lvm2 metadata - this is planned as we needed for other pieces.

So the issue is known - but since not many users ever reduce device size - it's low prio ATM - unless there is important case in mind behind where the manual 'workaround' is not good enough.
Comment 4 Eric Wheeler 2017-07-28 15:17:05 EDT
Interesting.

You might add an option for lvresize like `--discard-reduction` that would blkdiscard the tail.  This could work on traditional LVs as well, no reason for it not to: indeed, some users might wish to discard their shrunken LVs for SSDs and such.

Since the user is invoking it, they can be aware of the potentially long resize/lock times by warning them in the option documentation.
Comment 5 Zdenek Kabelac 2017-07-28 15:26:30 EDT
For normal 'case'  i.e. linear  LV -  reduced/released extents can be immediately discarded by using lvm.conf   issue_discard=1  option.

Using an option with lvresize/lvremove might be probably another way telling system user is 100% he will not want TRIMed data back.

So probaly woth something thinking.

Note You need to log in before you can comment on or make changes to this bug.