Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1622031

Summary: locking error when stacking thin snapshots
Product: Red Hat Enterprise Linux 7 Reporter: Roman Bednář <rbednar>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
lvm2 sub component: Thin Provisioning QA Contact: cluster-qe <cluster-qe>
Status: CLOSED NOTABUG Docs Contact:
Severity: medium    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, prockai, thornber, zkabelac
Version: 7.6Keywords: Regression
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-08-27 14:20:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
lvcreate_vvvv
none
vgcfgbackup
none
lvcreate_vvvv_RHEL75
none
do not use
none
do not use
none
lvcreate_vvvv none

Description Roman Bednář 2018-08-24 08:10:19 UTC
Reproducible with clvmd cluster.

SCENARIO - [stacked_thin_snap_force_removal]
 Stack a snapshot on top of an existing snapshot then remove the PV LV out from under LVM
 Setting up base level origin/snapshot
 Making pool volume
 lvcreate  --thinpool POOL -L 1G  --zero y --poolmetadatasize 4M snapper_thinp
 
 Sanity checking pool device (POOL) metadata
 thin_check /dev/mapper/snapper_thinp-meta_swap.856
 examining superblock
 examining devices tree
 examining mapping tree
 checking space map counts
 
 Making origin volume
 lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
 lvcreate  -V 1G -T snapper_thinp/POOL -n other1
   WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other2
   WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other3
   WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other4
   WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate  -V 1G -T snapper_thinp/POOL -n other5
   WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
 lvcreate  -y -k n -s /dev/snapper_thinp/origin -n snap_level1
 Creating stacked level PV/VG
 pvcreate -y /dev/snapper_thinp/snap_level1
 vgcreate  -f snapper_thinp_stack /dev/snapper_thinp/snap_level1
 Creating stacked level origin/snapshot
 lvcreate  -L 100M snapper_thinp_stack -n origin
   Error locking on node 2: Reading VG not found for LVID XqX55FQvnd7ODuOgCu5t4POyLStRHKhFdZCqNScZeHf4dvzJgJCAjLtluvXLyeOc
   Failed to activate new LV.
 couldn't create stacked origin



lvm2-2.02.180-5

Comment 2 Roman Bednář 2018-08-27 10:03:37 UTC
Tested today with 7.5 and passed. This concrete scenario was added in 7.6 but there is older one similar to this that's testing snapshot stacking, which is passing with 7.5 but fails with locking error in 7.6. This indicates a regression.


7.5 pass:
https://beaker.cluster-qe.lab.eng.brq.redhat.com/bkr/jobs/86272

SCENARIO - [stacked_thin_snaps]
Stack snapshots on top of existing snapshots
Setting up base level origin/snapshot
Making pool volume
lvcreate --activate ey --thinpool POOL -L 1G  --zero n --poolmetadatasize 4M snapper_thinp

Sanity checking pool device (POOL) metadata
thin_check /dev/mapper/snapper_thinp-meta_swap.885
examining superblock
examining devices tree
examining mapping tree
checking space map counts


Making origin volume
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey --virtualsize 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate --activate ey -y -k n -s /dev/snapper_thinp/origin -n snap_level1
Creating stacked level PV/VG
Creating stacked level origin/snapshot
lvcreate --activate ey -L 100M snapper_thinp_stack -n origin
lvcreate --activate ey -s /dev/snapper_thinp_stack/origin -n snap_level2 -L 50M
Removing stacked level origin/snapshot
Removing stacked level VG/PV
Removing base level VG/PV
Removing snap volume snapper_thinp/snap_level1
lvremove -f /dev/snapper_thinp/snap_level1
Removing thin origin and other virtual thin volumes
Removing pool snapper_thinp/POOL


lvm2-2.02.177-4.el7.x86_64



=======================

7.6 fail:
https://beaker.cluster-qe.lab.eng.brq.redhat.com/bkr/jobs/86275


SCENARIO - [stacked_thin_snaps]
Stack snapshots on top of existing snapshots
Setting up base level origin/snapshot
Making pool volume
lvcreate  --thinpool POOL -L 1G --profile thin-performance --zero y --poolmetadatasize 4M snapper_thinp

Sanity checking pool device (POOL) metadata
thin_check /dev/mapper/snapper_thinp-meta_swap.759
examining superblock
examining devices tree
examining mapping tree
checking space map counts

Making origin volume
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other1
  WARNING: Sum of all thin volume sizes (2.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other2
  WARNING: Sum of all thin volume sizes (3.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate  -V 1G -T snapper_thinp/POOL -n other3
  WARNING: Sum of all thin volume sizes (4.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate  -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (1.00 GiB).
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n snap_level1
Creating stacked level PV/VG
pvcreate -y /dev/snapper_thinp/snap_level1
vgcreate  -f snapper_thinp_stack /dev/snapper_thinp/snap_level1
Creating stacked level origin/snapshot
lvcreate  -L 100M snapper_thinp_stack -n origin
  Error locking on node 2: Reading VG not found for LVID gz7Kfku214KFWCOImjOqvgyvUwS90lN7Pw4XSkB17OjBBho7yvfGd8w4m7N3Gi1b
  Failed to activate new LV.
couldn't create stacked origin


lvm2-2.02.180-5.el7.x86_64

Comment 4 Roman Bednář 2018-08-27 13:02:45 UTC
Created attachment 1478946 [details]
lvcreate_vvvv

Comment 5 Roman Bednář 2018-08-27 13:03:17 UTC
Created attachment 1478947 [details]
vgcfgbackup

Comment 6 Roman Bednář 2018-08-27 13:18:14 UTC
Created attachment 1478950 [details]
lvcreate_vvvv_RHEL75

Comment 7 Roman Bednář 2018-08-27 13:44:44 UTC
Created attachment 1478957 [details]
do not use

removing incorrect log

Comment 8 Zdenek Kabelac 2018-08-27 13:46:56 UTC
(In reply to Roman Bednář from comment #4)
> Created attachment 1478946 [details]
> lvcreate_vvvv

Yep - this seems to work as expected.

ThinLV is made active/present only on 1 node in cluster - while lvcreate on top of this thinLV requests cluster-wide activation, but since other cluster nodes do NOT see this thinLV as PV (not available to them) - it fails on error where remote clvmd cannot find this PV/VG present.

Correct usage would be  'lvcreate -aey' for this origin creation.

So far it more or less look like a bugfix.

Comment 9 Roman Bednář 2018-08-27 13:49:32 UTC
Created attachment 1478962 [details]
do not use

Comment 10 Roman Bednář 2018-08-27 13:52:49 UTC
Created attachment 1478963 [details]
lvcreate_vvvv

adding correct -vvvv logs

Comment 11 Zdenek Kabelac 2018-08-27 14:09:41 UTC
(In reply to Roman Bednář from comment #10)
> Created attachment 1478963 [details]
> lvcreate_vvvv
> 
> adding correct -vvvv logs

So here both logs for 7.5 & 7.6 seems to match - and complain about locking error.

Comment 12 Roman Bednář 2018-08-27 14:20:12 UTC
Closing. This is a test issue. We're missing the needed exclusive activation flag in 7.6.