RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1906412 - "Detach cachevol before removing" error when attempting to uncache writecache pool
Summary: "Detach cachevol before removing" error when attempting to uncache writecache...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.4
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-10 13:19 UTC by Corey Marthaler
Modified: 2021-09-07 11:49 UTC (History)
8 users (show)

Fixed In Version: lvm2-2.03.11-0.4.20201222gitb84a992.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-05-18 15:02:04 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
verbose uncache attempt (178.47 KB, text/plain)
2020-12-10 13:20 UTC, Corey Marthaler 		no flags Details
View All

Description Corey Marthaler 2020-12-10 13:19:18 UTC 
Description of problem:
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sde1
*  pool (fast):    /dev/sdd1
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=44 high_watermark=93 writeback_jobs=2213 autocommit_blocks=2352 autocommit_time=2242' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin


Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Waiting for the snap merge to complete...
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin


Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin
couldn't uncache/remove cache pool volume



14:07:16.398100 lvconvert[125988] device_mapper/libdm-common.c:1496  writecache_sanity-cworigin: Processing NODE_READ_AHEAD 8192 (flags=1)
14:07:16.398116 lvconvert[125988] device_mapper/libdm-common.c:1250  writecache_sanity-cworigin (253:7): read ahead is 8192
14:07:16.398122 lvconvert[125988] device_mapper/libdm-common.c:1375  writecache_sanity-cworigin: retaining kernel read ahead of 8192 (requested 8192)
14:07:16.398131 lvconvert[125988] activate/dev_manager.c:817  Getting device info for writecache_sanity-cworigin [LVM-YwD3uKdd3DJtAi0EUQxbnVuwuviieQGOuRBLfNDpDEZPLBvhozvbrm8eBn3cslIj].
14:07:16.398149 lvconvert[125988] device_mapper/ioctl/libdm-iface.c:1898  dm status  LVM-YwD3uKdd3DJtAi0EUQxbnVuwuviieQGOuRBLfNDpDEZPLBvhozvbrm8eBn3cslIj [ opencount noflush ]   [16384] (*1)
14:07:16.398173 lvconvert[125988] device_mapper/libdm-common.c:1250  writecache_sanity-cworigin (253:7): read ahead is 8192
14:07:16.398198 lvconvert[125988] metadata/lv_manip.c:6637  Detach cachevol before removing.




Version-Release number of selected component (if applicable):
kernel-4.18.0-259.el8.dt2    BUILT: Mon Dec  7 22:20:12 CET 2020
lvm2-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 14:04:46 CET 2020
lvm2-libs-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 14:04:46 CET 2020


How reproducible:
Often
Comment 1 Corey Marthaler 2020-12-10 13:20:38 UTC 
Created attachment 1738192 [details]
verbose uncache attempt
Comment 2 David Teigland 2020-12-10 21:47:31 UTC 
fix https://sourceware.org/git/?p=lvm2.git;a=commit;h=5dbe2fdd9dd783c2911a5375d6fdec86118af280

Note this is a fix for an 8.4 feature being added in bug 1805896.
Comment 5 Corey Marthaler 2021-02-01 23:27:40 UTC 
Moving to Verified:Tested based on results with the following build.

kernel-4.18.0-277.el8    BUILT: Wed Jan 20 09:06:28 CST 2021
lvm2-2.03.11-1.el8    BUILT: Fri Jan  8 05:21:07 CST 2021
lvm2-libs-2.03.11-1.el8    BUILT: Fri Jan  8 05:21:07 CST 2021



============================================================
Iteration 10 of 10 started at Thu Jan 28 18:34:19 CST 2021
============================================================
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdf
*  pool (fast):    /dev/sdg
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=25 high_watermark=51 writeback_jobs=2914 autocommit_blocks=1920 autocommit_time=2549' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin


Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin
Although the activate passed, errors were found in its output
LINE=18:39:33.443772 lvchange[309768] label/label.c:824  Scanned devices: read errors 0 process errors 0 failed 0

Although the activate passed, errors were found in its output
LINE=18:39:36.542565 lvchange[309768] device_mapper/libdm-deptree.c:3051  Adding target to (253:11): 0 8388608 error 

Although the activate passed, errors were found in its output
LINE=18:39:36.734025 lvchange[309768] notify/lvmnotify.c:69  D-Bus notification failed: The name com.redhat.lvmdbus1 was not provided by any .service files

Waiting for the snap merge to complete...
  Failed to find logical volume "writecache_sanity/merge"
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin


Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin
Removing cache origin volume writecache_sanity/cworigin
lvremove -f /dev/writecache_sanity/cworigin
Comment 9 Corey Marthaler 2021-02-15 20:08:22 UTC 
Final verification with the latest rpms.

kernel-4.18.0-283.el8    BUILT: Thu Feb  4 05:30:59 CST 2021

lvm2-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021
lvm2-libs-2.03.11-4.el8    BUILT: Thu Feb 11 04:35:23 CST 2021


============================================================
Iteration 10 of 10 started at Mon Feb 15 13:58:46 CST 2021
============================================================
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdi
*  pool (fast):    /dev/sde
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=48 high_watermark=94 writeback_jobs=1973 autocommit_blocks=1612 autocommit_time=1971' --cachevol writecache_sanity/pool writecache_sanity/cworigin

Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Waiting for the snap merge to complete...
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin

Uncaching cache origin (lvconvert --yes --uncache writecache_sanity/cworigin) from cache origin
Comment 11 errata-xmlrpc 2021-05-18 15:02:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659
Note You need to log in before you can comment on or make changes to this bug.