Bug 1906412

Summary: "Detach cachevol before removing" error when attempting to uncache writecache pool
Product: Red Hat Enterprise Linux 8 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: Cache Logical Volumes QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: agk, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, teigland, zkabelac
Version: 8.4   
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.03.11-0.4.20201222gitb84a992.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:02:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
verbose uncache attempt none

Description Corey Marthaler 2020-12-10 13:19:18 UTC
Description of problem:
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sde1
*  pool (fast):    /dev/sdd1
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=44 high_watermark=93 writeback_jobs=2213 autocommit_blocks=2352 autocommit_time=2242' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin


Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Waiting for the snap merge to complete...
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin


Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin
couldn't uncache/remove cache pool volume



14:07:16.398100 lvconvert[125988] device_mapper/libdm-common.c:1496  writecache_sanity-cworigin: Processing NODE_READ_AHEAD 8192 (flags=1)
14:07:16.398116 lvconvert[125988] device_mapper/libdm-common.c:1250  writecache_sanity-cworigin (253:7): read ahead is 8192
14:07:16.398122 lvconvert[125988] device_mapper/libdm-common.c:1375  writecache_sanity-cworigin: retaining kernel read ahead of 8192 (requested 8192)
14:07:16.398131 lvconvert[125988] activate/dev_manager.c:817  Getting device info for writecache_sanity-cworigin [LVM-YwD3uKdd3DJtAi0EUQxbnVuwuviieQGOuRBLfNDpDEZPLBvhozvbrm8eBn3cslIj].
14:07:16.398149 lvconvert[125988] device_mapper/ioctl/libdm-iface.c:1898  dm status  LVM-YwD3uKdd3DJtAi0EUQxbnVuwuviieQGOuRBLfNDpDEZPLBvhozvbrm8eBn3cslIj [ opencount noflush ]   [16384] (*1)
14:07:16.398173 lvconvert[125988] device_mapper/libdm-common.c:1250  writecache_sanity-cworigin (253:7): read ahead is 8192
14:07:16.398198 lvconvert[125988] metadata/lv_manip.c:6637  Detach cachevol before removing.




Version-Release number of selected component (if applicable):
kernel-4.18.0-259.el8.dt2    BUILT: Mon Dec  7 22:20:12 CET 2020
lvm2-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 14:04:46 CET 2020
lvm2-libs-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 14:04:46 CET 2020


How reproducible:
Often

Comment 1 Corey Marthaler 2020-12-10 13:20:38 UTC
Created attachment 1738192 [details]
verbose uncache attempt

Comment 2 David Teigland 2020-12-10 21:47:31 UTC
fix https://sourceware.org/git/?p=lvm2.git;a=commit;h=5dbe2fdd9dd783c2911a5375d6fdec86118af280

Note this is a fix for an 8.4 feature being added in bug 1805896.

Comment 5 Corey Marthaler 2021-02-01 23:27:40 UTC
Moving to Verified:Tested based on results with the following build.

kernel-4.18.0-277.el8    BUILT: Wed Jan 20 09:06:28 CST 2021
lvm2-2.03.11-1.el8    BUILT: Fri Jan  8 05:21:07 CST 2021
lvm2-libs-2.03.11-1.el8    BUILT: Fri Jan  8 05:21:07 CST 2021



============================================================
Iteration 10 of 10 started at Thu Jan 28 18:34:19 CST 2021
============================================================
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdf
*  pool (fast):    /dev/sdg
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=25 high_watermark=51 writeback_jobs=2914 autocommit_blocks=1920 autocommit_time=2549' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin


Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin
Although the activate passed, errors were found in its output
LINE=18:39:33.443772 lvchange[309768] label/label.c:824  Scanned devices: read errors 0 process errors 0 failed 0

Although the activate passed, errors were found in its output
LINE=18:39:36.542565 lvchange[309768] device_mapper/libdm-deptree.c:3051  Adding target to (253:11): 0 8388608 error 

Although the activate passed, errors were found in its output
LINE=18:39:36.734025 lvchange[309768] notify/lvmnotify.c:69  D-Bus notification failed: The name com.redhat.lvmdbus1 was not provided by any .service files

Waiting for the snap merge to complete...
  Failed to find logical volume "writecache_sanity/merge"
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin


Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin
Removing cache origin volume writecache_sanity/cworigin
lvremove -f /dev/writecache_sanity/cworigin

Comment 9 Corey Marthaler 2021-02-15 20:08:22 UTC
Final verification with the latest rpms.

kernel-4.18.0-283.el8    BUILT: Thu Feb  4 05:30:59 CST 2021

lvm2-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021
lvm2-libs-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021


============================================================
Iteration 10 of 10 started at Mon Feb 15 13:58:46 CST 2021
============================================================
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdi
*  pool (fast):    /dev/sde
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=48 high_watermark=94 writeback_jobs=1973 autocommit_blocks=1612 autocommit_time=1971' --cachevol writecache_sanity/pool writecache_sanity/cworigin

Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Waiting for the snap merge to complete...
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin

Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin

Comment 11 errata-xmlrpc 2021-05-18 15:02:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659