Bug 1661987
| Summary: | cannot remove cache lv when its larger than 1TB | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | nikhil kshirsagar <nkshirsa> | ||||
| Component: | lvm2 | Assignee: | Joe Thornber <thornber> | ||||
| lvm2 sub component: | Cache Logical Volumes | QA Contact: | cluster-qe <cluster-qe> | ||||
| Status: | CLOSED ERRATA | Docs Contact: | |||||
| Severity: | high | ||||||
| Priority: | high | CC: | agk, alex.wang, cmarthal, heinzm, jbrassow, lvm-team, mcsontos, msnitzer, prajnoha, rbednar, rhandlin, thornber, zkabelac | ||||
| Version: | 7.6 | Keywords: | ZStream | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | lvm2-2.02.184-1.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1696740 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-08-06 13:10:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1696740 | ||||||
| Attachments: |
|
||||||
Created attachment 1516656 [details]
verbose output of a failing lvremove
Issue recreated in dmtest (no LVM). This is a kernel bug. https://github.com/jthornber/device-mapper-test-suite/commit/9d79289896bc08bd9b67d69a6fc667819f18b8c8 There is a kernel parameter called the migration_threshold, that is used to throttle background IO that the dm-cache target performs (promotions, demotions and writebacks). In this case the migration_threshold set by LVM is smaller than a single block, so no writebacks occur. With a huge system like this they will certainly need a much higher migration threshold so they can actually populate the cache (as well as clean it). The quick fix for the customer is to change this parameter. Longer term I think there are a couple of things to consider: i) Should the kernel always allow one blocks worth of background IO? Effectively taking the migration_threshold = max(block_size, migration_threshold) ii) LVM should certainly set an initial migration threshold at least a equal to a block. FYI, lvm2 stable branch commit bc6ae7030ad3c620fadce2654c106f78a5bc2350 Fix verified in the latest rpms.
3.10.0-1048.el7.x86_64
lvm2-2.02.185-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-libs-2.02.185-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-cluster-2.02.185-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-lockd-2.02.185-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
lvm2-python-boom-0.9-17.el7.2 BUILT: Mon May 13 04:37:00 CDT 2019
cmirror-2.02.185-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-1.02.158-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-libs-1.02.158-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-1.02.158-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-event-libs-1.02.158-1.el7 BUILT: Mon May 13 04:36:30 CDT 2019
device-mapper-persistent-data-0.8.1-1.el7 BUILT: Sat May 4 14:53:53 CDT 2019
SCENARIO - [large_cache_removal]
Create a cache volume with a large pool (larger than 1TB if storage allows) with writeback mode and attempt to remove it w/o getting into a "Flushing blocks for cache" loop
Largest PV (/dev/sdg1) found is 476799, greater than 1T so this *is* a valid test for ths bug
*** Cache info for this scenario ***
* origin (slow): /dev/sdh1 /dev/sde1 /dev/sdd1
* pool (fast): /dev/sdg1 /dev/sdb1
************************************
Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate -i 3 -L 2G -n large_cache_removal cache_sanity @slow
lvcreate -l 100%PVS -n pool cache_sanity /dev/sdg1
lvcreate -L 2G -n pool_meta cache_sanity @fast
WARNING: No free extents on physical volume "/dev/sdg1".
Create cache pool volume by combining the cache data and cache metadata (fast) volumes with WRITEBACK mode
lvconvert --yes --type cache-pool --cachemode writeback --poolmetadata cache_sanity/pool_meta cache_sanity/pool
WARNING: Converting cache_sanity/pool and cache_sanity/pool_meta to cache pool's data and metadata volumes with metadata wiping.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Create cached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type cache --cachepool cache_sanity/pool cache_sanity/large_cache_removal
Placing an xfs filesystem on origin volume
Mounting origin volume
Writing files to /mnt/large_cache_removal
Checking files on /mnt/large_cache_removal
lvremove -f /dev/cache_sanity/large_cache_removal
(this should not loop "Flushing blocks for cache cache_sanity/large_cache_removal")
Unknown feature in status: 8 2019/524288 3968 89/984359 173 40 3215 0 0 89 77 3 metadata2 writeback no_discard_passdown 2 migration_threshold 31744 smq 0 rw -
Flushing 77 blocks for cache cache_sanity/large_cache_removal.
Flushing 74 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 74 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 69 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 69 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 69 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 69 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 69 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 54 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 54 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 42 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 42 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 28 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 28 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 17 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 17 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Flushing 5 blocks for cache cache_sanity/large_cache_removal.
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 5 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Unknown feature in status: 8 2995/524288 3968 89/984359 39 0 0 0 0 0 0 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 31744 cleaner 0 rw -
Logical volume "pool" successfully removed
Logical volume "large_cache_removal" successfully removed
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2253 |
Description of problem: On RHEL 7.6, we cannot remove cache LV when created with policy writeback (no problem with writethrough) Version-Release number of selected component (if applicable): 3.10.0-957.1.3.el7.x86_64 lvm2-2.02.180-10.el7_6.2.x86_64 lvm2-libs-2.02.180-10.el7_6.2.x86_64 How reproducible: [root@vm250-56 ~]# vgcreate test /dev/vdj Volume group "test" successfully created [root@vm250-56 ~]# lvcreate -L1.02T -n test test /dev/vdj Rounding up size to full physical extent 1.02 TiB WARNING: xfs signature detected on /dev/test/test at offset 0. Wipe it? [y/n]: y Wiping xfs signature on /dev/test/test. Logical volume "test" created. [root@vm250-56 ~]# vgextend test /dev/vdi3 Volume group "test" successfully extended [root@vm250-56 ~]# lvcreate -L 2G -n test_cache_meta test /dev/vdi3 Logical volume "test_cache_meta" created. [root@vm250-56 ~]# vgextend test /dev/vdk Volume group "test" successfully extended [root@vm250-56 ~]# lvcreate -L 1.02T -n test_cache test /dev/vdk <---- Rounding up size to full physical extent 1.02 TiB Logical volume "test_cache" created. [root@vm250-56 ~]# lvconvert --type cache-pool --cachemode writeback --poolmetadata test/test_cache_meta test/test_cache Using 1.09 MiB chunk size instead of default 64.00 KiB, so cache pool has less than 1000000 chunks. WARNING: Converting test/test_cache and test/test_cache_meta to cache pool's data and metadata volumes with metadata wiping. THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.) Do you really want to convert test/test_cache and test/test_cache_meta? [y/n]: y Converted test/test_cache and test/test_cache_meta to cache pool. [root@vm250-56 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel_vm253-69 -wi-ao---- <13.91g swap rhel_vm253-69 -wi-ao---- 1.60g [lvol0_pmspare] test ewi------- 2.00g test test -wi-a----- 1.02t test_cache test Cwi---C--- 1.02t [test_cache_cdata] test Cwi------- 1.02t [test_cache_cmeta] test ewi------- 2.00g vdo_lv vdo_vg -wi-ao---- 10.00g [root@vm250-56 ~]# lvconvert --type cache --cachepool test/test_cache test/test Do you want wipe existing metadata of cache pool test/test_cache? [y/n]: y Logical volume test/test is now cached. [root@vm250-56 ~]# lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel_vm253-69 -wi-ao---- <13.91g swap rhel_vm253-69 -wi-ao---- 1.60g [lvol0_pmspare] test ewi------- 2.00g test test Cwi-a-C--- 1.02t [test_cache] [test_corig] 0.00 0.38 0.00 [test_cache] test Cwi---C--- 1.02t 0.00 0.38 0.00 [test_cache_cdata] test Cwi-ao---- 1.02t [test_cache_cmeta] test ewi-ao---- 2.00g [test_corig] test owi-aoC--- 1.02t vdo_lv vdo_vg -wi-ao---- 10.00g [root@vm250-56 ~]# mkfs.xfs /dev/test/test meta-data=/dev/test/test isize=512 agcount=32, agsize=8556520 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=273804288, imaxpct=5 = sunit=280 swidth=280 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=133696, version=2 = sectsz=512 sunit=8 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@vm250-56 ~]# [root@vm250-56 ~]# [root@vm250-56 ~]# lvremove test/test Do you really want to remove active logical volume test/test? [y/n]: y Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. Flushing 191 blocks for cache test/test. ^C Interrupted... Flushing of test/test aborted. Failed to uncache test/test. Additional info: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878441