RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1661987 - cannot remove cache lv when its larger than 1TB
Summary: cannot remove cache lv when its larger than 1TB
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: lvm2
Version: 7.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Joe Thornber
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1696740
TreeView+ depends on / blocked
 
Reported: 2018-12-25 07:02 UTC by nikhil kshirsagar
Modified: 2021-09-03 12:49 UTC (History)
13 users (show)

Fixed In Version: lvm2-2.02.184-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1696740 (view as bug list)
Environment:
Last Closed: 2019-08-06 13:10:41 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
verbose output of a failing lvremove (185.05 KB, text/plain)
2018-12-25 13:35 UTC, nikhil kshirsagar 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:2253 0 None None None 2019-08-06 13:11:12 UTC

Description nikhil kshirsagar 2018-12-25 07:02:50 UTC 
Description of problem:
On RHEL 7.6, we cannot remove cache LV when created with policy writeback
(no problem with writethrough)

Version-Release number of selected component (if applicable):
3.10.0-957.1.3.el7.x86_64

lvm2-2.02.180-10.el7_6.2.x86_64
lvm2-libs-2.02.180-10.el7_6.2.x86_64


How reproducible:


[root@vm250-56 ~]# vgcreate test /dev/vdj
  Volume group "test" successfully created
[root@vm250-56 ~]# lvcreate -L1.02T -n test test /dev/vdj
  Rounding up size to full physical extent 1.02 TiB
WARNING: xfs signature detected on /dev/test/test at offset 0. Wipe it? [y/n]: y
  Wiping xfs signature on /dev/test/test.
  Logical volume "test" created.
[root@vm250-56 ~]# vgextend test /dev/vdi3
  Volume group "test" successfully extended
[root@vm250-56 ~]#  lvcreate -L 2G -n test_cache_meta test /dev/vdi3
  Logical volume "test_cache_meta" created.
[root@vm250-56 ~]# vgextend test /dev/vdk
  Volume group "test" successfully extended
[root@vm250-56 ~]#  lvcreate -L 1.02T -n test_cache test /dev/vdk <----
  Rounding up size to full physical extent 1.02 TiB
  Logical volume "test_cache" created.
[root@vm250-56 ~]# lvconvert --type cache-pool --cachemode writeback --poolmetadata test/test_cache_meta test/test_cache
  Using 1.09 MiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks.
  WARNING: Converting test/test_cache and test/test_cache_meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert test/test_cache and test/test_cache_meta? [y/n]: y
  Converted test/test_cache and test/test_cache_meta to cache pool.
[root@vm250-56 ~]# lvs -a
  LV                 VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root               rhel_vm253-69 -wi-ao---- <13.91g                                                    
  swap               rhel_vm253-69 -wi-ao----   1.60g                                                    
  [lvol0_pmspare]    test          ewi-------   2.00g                                                    
  test               test          -wi-a-----   1.02t                                                    
  test_cache         test          Cwi---C---   1.02t                                                    
  [test_cache_cdata] test          Cwi-------   1.02t                                                    
  [test_cache_cmeta] test          ewi-------   2.00g                                                    
  vdo_lv             vdo_vg        -wi-ao----  10.00g                                                    
[root@vm250-56 ~]# lvconvert --type cache --cachepool test/test_cache test/test
Do you want wipe existing metadata of cache pool test/test_cache? [y/n]: y
  Logical volume test/test is now cached.
[root@vm250-56 ~]# lvs -a
  LV                 VG            Attr       LSize   Pool         Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  root               rhel_vm253-69 -wi-ao---- <13.91g                                                                  
  swap               rhel_vm253-69 -wi-ao----   1.60g                                                                  
  [lvol0_pmspare]    test          ewi-------   2.00g                                                                  
  test               test          Cwi-a-C---   1.02t [test_cache] [test_corig] 0.00   0.38            0.00            
  [test_cache]       test          Cwi---C---   1.02t                           0.00   0.38            0.00            
  [test_cache_cdata] test          Cwi-ao----   1.02t                                                                  
  [test_cache_cmeta] test          ewi-ao----   2.00g                                                                  
  [test_corig]       test          owi-aoC---   1.02t                                                                  
  vdo_lv             vdo_vg        -wi-ao----  10.00g                                                                  
[root@vm250-56 ~]# mkfs.xfs /dev/test/test
meta-data=/dev/test/test         isize=512    agcount=32, agsize=8556520 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=273804288, imaxpct=5
         =                       sunit=280    swidth=280 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=133696, version=2
         =                       sectsz=512   sunit=8 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@vm250-56 ~]# 
[root@vm250-56 ~]# 
[root@vm250-56 ~]# lvremove test/test
Do you really want to remove active logical volume test/test? [y/n]: y
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
  Flushing 191 blocks for cache test/test.
^C  Interrupted...
  Flushing of test/test aborted.
  Failed to uncache test/test.



Additional info:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878441
Comment 2 nikhil kshirsagar 2018-12-25 13:35:43 UTC 
Created attachment 1516656 [details]
verbose output of a failing lvremove
Comment 8 Joe Thornber 2019-01-07 14:32:46 UTC 
Issue recreated in dmtest (no LVM).  This is a kernel bug.

https://github.com/jthornber/device-mapper-test-suite/commit/9d79289896bc08bd9b67d69a6fc667819f18b8c8
Comment 9 Joe Thornber 2019-01-08 13:02:18 UTC 
There is a kernel parameter called the migration_threshold, that is used to throttle background IO that
the dm-cache target performs (promotions, demotions and writebacks).

In this case the migration_threshold set by LVM is smaller than a single block, so no writebacks occur.

With a huge system like this they will certainly need a much higher migration threshold so they can actually populate the cache (as well as clean it).

The quick fix for the customer is to change this parameter.


Longer term I think there are a couple of things to consider:

i) Should the kernel always allow one blocks worth of background IO?  Effectively taking
   the migration_threshold = max(block_size, migration_threshold)

ii) LVM should certainly set an initial migration threshold at least a equal to a block.
Comment 13 Marian Csontos 2019-03-27 14:48:52 UTC 
FYI, lvm2 stable branch commit bc6ae7030ad3c620fadce2654c106f78a5bc2350
Comment 16 Corey Marthaler 2019-05-16 20:55:35 UTC 
Fix verified in the latest rpms.

3.10.0-1048.el7.x86_64

lvm2-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-libs-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-cluster-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-lockd-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-python-boom-0.9-17.el7.2    BUILT: Mon May 13 04:37:00 CDT 2019
cmirror-2.02.185-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-libs-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-libs-1.02.158-1.el7    BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-persistent-data-0.8.1-1.el7    BUILT: Sat May  4 14:53:53 CDT 2019



SCENARIO - [large_cache_removal]
Create a cache volume with a large pool (larger than 1TB if storage allows) with writeback mode and attempt to remove it w/o getting into a "Flushing blocks for cache" loop
Largest PV (/dev/sdg1) found is 476799, greater than 1T so this *is* a valid test for ths bug

*** Cache info for this scenario ***
*  origin (slow):  /dev/sdh1 /dev/sde1 /dev/sdd1
*  pool (fast):    /dev/sdg1 /dev/sdb1
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate  -i 3 -L 2G -n large_cache_removal cache_sanity @slow

lvcreate  -l 100%PVS -n pool cache_sanity /dev/sdg1
lvcreate  -L 2G -n pool_meta cache_sanity @fast
  WARNING: No free extents on physical volume "/dev/sdg1".

Create cache pool volume by combining the cache data and cache metadata (fast) volumes with WRITEBACK mode
lvconvert --yes --type cache-pool --cachemode writeback --poolmetadata cache_sanity/pool_meta cache_sanity/pool
  WARNING: Converting cache_sanity/pool and cache_sanity/pool_meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Create cached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type cache --cachepool cache_sanity/pool cache_sanity/large_cache_removal
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/large_cache_removal

Checking files on /mnt/large_cache_removal

lvremove -f /dev/cache_sanity/large_cache_removal
        (this should not loop "Flushing blocks for cache cache_sanity/large_cache_removal")

  Unknown feature in status: 8 2019/524288 3968 89/984359 173 40 3215 0 0 89 77 3 metadata2 writeback no_discard_passdown 2 migration_threshold 31744 smq 0 rw - 
  Flushing 77 blocks for cache cache_sanity/large_cache_removal.
  Flushing 74 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 74 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 69 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 69 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 69 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 69 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 54 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 54 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 42 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 42 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 28 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 28 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 17 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 17 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Flushing 5 blocks for cache cache_sanity/large_cache_removal.
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 5 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 0 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw - 
  Logical volume "pool" successfully removed
  Logical volume "large_cache_removal" successfully removed
Comment 18 errata-xmlrpc 2019-08-06 13:10:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2253
Note You need to log in before you can comment on or make changes to this bug.