Bug 1854558

Summary: snapshots of writecache volumes have been enabled
Product: Red Hat Enterprise Linux 8 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: David Teigland <teigland>
lvm2 sub component: Cache Logical Volumes QA Contact: cluster-qe <cluster-qe>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: high CC: agk, heinzm, jbrassow, mcsontos, msnitzer, pasik, prajnoha, teigland, zkabelac
Version: 8.3   
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.03.11-0.2.20201103git8801a86.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:01:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2020-07-07 16:42:13 UTC
Description of problem:
As a part of 8.2 writecache testing, we had to verify that snapshots of writecache volumes were not allowed (bug 1798631). It appears this was just re-enabled in the latest lvm build, but i don't believe we were told to enable testing of this yet. Is this feature supposed to be turned on now?

8.2:
[root@hayes-01 ~]# lvcreate  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
  Snapshots of writecache are not supported.


8.3:
[root@hayes-03 ~]# lvcreate  -s /dev/writecache_sanity/cworigin -c 64 -n merge1 -L 500M
  Logical volume "merge1" created.


Version-Release number of selected component (if applicable):
kernel-4.18.0-221.el8    BUILT: Thu Jun 25 16:28:39 CDT 2020
lvm2-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-libs-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
lvm2-dbusd-2.03.09-3.el8    BUILT: Mon Jun 29 13:53:38 CDT 2020
lvm2-lockd-2.03.09-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020
device-mapper-event-libs-1.02.171-3.el8    BUILT: Mon Jun 29 13:50:23 CDT 2020

Comment 1 David Teigland 2020-07-07 17:08:56 UTC
The snapshot bug was caused by using the wrong block size which is now fixed (like allowing writecache attach to an active LV.)

Comment 2 Jonathan Earl Brassow 2020-07-15 05:54:52 UTC
(In reply to David Teigland from comment #1)
> The snapshot bug was caused by using the wrong block size which is now fixed
> (like allowing writecache attach to an active LV.)

So snapshots have been enabled then?  Might want to negotiate that feature with Corey...

Comment 3 David Teigland 2020-07-15 17:23:47 UTC
The implication of fixing the snapshot bug was that we enable snapshots.  If we don't want to enable them for other reasons, then that's fine.

Comment 4 Corey Marthaler 2020-07-15 19:12:43 UTC
I'm fine keeping them enabled and have the snapshot tests again turned on. 

The issue for QA however, was that there was no bug in the current errata that mentioned anything about snapshots or a snapshot bug being fixed. Was this fix added as a part of bug 1630978 or rebase bug 1824209?

Comment 5 David Teigland 2020-07-15 19:21:12 UTC
bug 1630978 comment 6 lists three commits which add fs block size detection (lack of this caused the problem) and enable snapshots.

Comment 10 Corey Marthaler 2020-11-19 03:11:16 UTC
Fix verified in the latest .d rpms.

kernel-4.18.0-246.el8.dt2    BUILT: Mon Nov  9 07:22:41 CST 2020
lvm2-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020
lvm2-libs-2.03.11-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020

device-mapper-1.02.175-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020
device-mapper-libs-1.02.175-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020
device-mapper-event-1.02.175-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020
device-mapper-event-libs-1.02.175-0.2.20201103git8801a86.el8    BUILT: Wed Nov  4 07:04:46 CST 2020



SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdg1
*  pool (fast):    /dev/sdh1
************************************

Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=6 high_watermark=84 writeback_jobs=2491 autocommit_blocks=1279 autocommit_time=2334' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin

Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )
lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin

Comment 14 Corey Marthaler 2021-01-11 18:32:24 UTC
Fix verified in the latest rpms.

kernel-4.18.0-271.el8    BUILT: Fri Jan  8 03:32:43 CST 2021
lvm2-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020
lvm2-libs-2.03.11-0.4.20201222gitb84a992.el8    BUILT: Tue Dec 22 06:33:49 CST 2020



============================================================
Iteration 4 of 4 started at Mon Jan 11 12:16:57 CST 2021
============================================================
SCENARIO - [simple_writecache_snap_merge]
Create snaps of writecache origin with fs data, verify data on snaps, change data on origin, merge data back to origin, verify origin data

*** Writecache info for this scenario ***
*  origin (slow):  /dev/sdd1
*  pool (fast):    /dev/sdc1
************************************

ls: cannot access '/dev/writecache_sanity': No such file or directory
Adding "slow" and "fast" tags to corresponding pvs
Create origin (slow) volume
lvcreate --yes --wipesignatures y  -L 4G -n cworigin writecache_sanity @slow

Create writecache cvol (fast) volumes
lvcreate --yes  -L 2G -n pool writecache_sanity @fast

Deactivate *ONLY* fast pool before conversion to write cache (SEE bug 1185347)
Create writecached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type writecache --cachesettings 'low_watermark=32 high_watermark=50 writeback_jobs=1125 autocommit_blocks=2399 autocommit_time=2364' --cachevol writecache_sanity/pool writecache_sanity/cworigin
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/cworigin

Checking files on /mnt/cworigin

Making a snapshot of the origin volume, mounting, and verifying original data
( ** This is again enabled in rhel8.3 ** )

lvcreate --yes  -s /dev/writecache_sanity/cworigin -c 64 -n merge -L 500M
+++ Mounting and verifying snapshot merge data +++
Checking files on /mnt/merge

Writing new origin data and then merging back the snapshot volume
Writing files to /mnt/cworigin
Checking files on /mnt/cworigin

Umount origin volume
Deactivating volume: cworigin
Merge snapshot writecache_sanity/merge back into the origin
lvconvert --yes --merge writecache_sanity/merge
Activating volume: cworigin

Waiting for the snap merge to complete...
Mount and verify the proper data now exists on the origin
Checking files on /mnt/cworigin

Separating cache pool (lvconvert --yes --splitcache writecache_sanity/cworigin) from cache origin
Removing cache pool writecache_sanity/pool
Removing cache origin volume writecache_sanity/cworigin
lvremove -f /dev/writecache_sanity/cworigin

Comment 16 errata-xmlrpc 2021-05-18 15:01:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1659