Bug 738435 - "Failed to refresh" error when attempting exclusive snap deletion from cluster VG on non active node
"Failed to refresh" error when attempting exclusive snap deletion from cluste...
Status: ASSIGNED
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.2
x86_64 Linux
low Severity medium
: rc
: ---
Assigned To: Alasdair Kergon
Corey Marthaler
:
Depends On:
Blocks: 756082
  Show dependency treegraph
 
Reported: 2011-09-14 15:05 EDT by Corey Marthaler
Modified: 2016-09-01 05:17 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2011-09-14 15:05:13 EDT
Description of problem:

# Exclusively activate a clustered VG and create a snapshot on one node, then attempt to remove the snapshot from another node.
[root@taft-03 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M      8   2011-09-14 13:43:48  taft-01
   2   M     12   2011-09-14 13:43:48  taft-02
   3   M      8   2011-09-14 13:43:48  taft-03
   4   M     16   2011-09-14 13:43:48  taft-04
[root@taft-03 ~]# lvcreate -n origin -L 100M taft
  Logical volume "origin" created
[root@taft-03 ~]# vgchange -an
  0 logical volume(s) in volume group "taft" now active
[root@taft-03 ~]# lvs
  LV      VG        Attr   LSize   Origin Snap%  
  origin  taft      -wi--- 100.00m             
[root@taft-03 ~]# vgchange -aye taft
  1 logical volume(s) in volume group "taft" now active
[root@taft-03 ~]# lvs -a -o +devices
  LV      VG        Attr   LSize   Origin Snap% Devices         
  origin  taft      -wi-a- 100.00m              /dev/sdb1(0)    
[root@taft-03 ~]# lvcreate -s taft/origin -n snap1 -L 12M 
  Logical volume "snap1" created
[root@taft-03 ~]# lvcreate -s taft/origin -n snap2 -L 12M 
  Logical volume "snap2" created
[root@taft-03 ~]# lvs -a -o +devices
  LV      VG        Attr   LSize   Origin Snap%  Devices         
  origin  taft      owi-a- 100.00m               /dev/sdb1(0)    
  snap1   taft      swi-a-  12.00m origin   0.00 /dev/sdb1(25)   
  snap2   taft      swi-a-  12.00m origin   0.00 /dev/sdb1(28)   
[root@taft-03 ~]# pvscan
  PV /dev/sdb1   VG taft        lvm2 [135.66 GiB / 135.54 GiB free]
  PV /dev/sdc1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sdd1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sde1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sdf1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sdg1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]
  PV /dev/sdh1   VG taft        lvm2 [135.66 GiB / 135.66 GiB free]


# Do the delete attempt of the snapshot from another node in the cluster where the VG isn't active.
[root@taft-04 ~]# lvs -a -o +devices
  LV      VG        Attr   LSize   Origin Snap%  Devices         
  origin  taft      owi--- 100.00m               /dev/sdb1(0)    
  snap1   taft      swi---  12.00m origin        /dev/sdb1(25)   
  snap2   taft      swi---  12.00m origin        /dev/sdb1(28)   
[root@taft-04 ~]# lvremove taft/snap2
Do you really want to remove active clustered logical volume snap2? [y/n]: y
  cluster request failed: Resource temporarily unavailable
  Failed to refresh origin without snapshot.
  Device '/dev/sdf1' has been left open.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdg1' has been left open.
  Device '/dev/sdh1' has been left open.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdd1' has been left open.
  Device '/dev/sdg1' has been left open.
  Device '/dev/sde1' has been left open.
  Device '/dev/sdf1' has been left open.
  Device '/dev/sdg1' has been left open.
  Device '/dev/sde1' has been left open.
  Device '/dev/sdh1' has been left open.
  Device '/dev/sdd1' has been left open.
  Device '/dev/sdf1' has been left open.
  Device '/dev/sdh1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sde1' has been left open.
  Device '/dev/sdg1' has been left open.
  Device '/dev/sdd1' has been left open.
  Device '/dev/sde1' has been left open.
  Device '/dev/sdf1' has been left open.
  Device '/dev/sdh1' has been left open.
  Device '/dev/sde1' has been left open.
  Device '/dev/sdd1' has been left open.
  Device '/dev/sdg1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdh1' has been left open.
  Device '/dev/sdf1' has been left open.
  Device '/dev/sdb1' has been left open.
  Device '/dev/sdc1' has been left open.
  Device '/dev/sdd1' has been left open.
[root@taft-04 ~]# lvs -a -o +devices
  LV      VG        Attr   LSize   Origin Snap%  Devices         
  origin  taft      owi--- 100.00m               /dev/sdb1(0)    
  snap1   taft      swi---  12.00m origin        /dev/sdb1(25)   
  snap2   taft      swi---  12.00m origin        /dev/sdb1(28)   



Version-Release number of selected component (if applicable):
2.6.32-195.el6.x86_64

lvm2-2.02.87-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
lvm2-libs-2.02.87-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
lvm2-cluster-2.02.87-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
udev-147-2.38.el6    BUILT: Fri Sep  9 16:25:50 CDT 2011
device-mapper-1.02.66-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
device-mapper-libs-1.02.66-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
device-mapper-event-1.02.66-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
device-mapper-event-libs-1.02.66-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011
cmirror-2.02.87-2.1.el6    BUILT: Wed Sep 14 09:44:16 CDT 2011


How reproducible:
Every time
Comment 1 Milan Broz 2011-09-14 15:47:17 EDT
One addition:
clvmd after restart read initial lock state from "lvs" command.

For virtual origin snapshot it loads wrong lock id (because virtual origin is internally represented by different LV).
This must be fixed as well to make it work reliably.
Comment 2 Alasdair Kergon 2012-01-04 16:34:12 EST
We should attempt to clean up the way it handles these sorts of situations.

(We probably can't support deleting a snapshot from a node different from the one where it's active, though.)
Comment 4 Alasdair Kergon 2012-03-13 13:59:15 EDT
(We should perhaps consider this alongside the similar thin pool/volume activation questions.)

1.  Maybe it's time to change the way locking is handled with volumes activated implicitly.  Always hold locks for every active volume and have the code know how to acquire the extra locks?

2.  See whether this lvremove sequence can be made to work correctly in a cluster - or if not, detect and ban it.

Note You need to log in before you can comment on or make changes to this bug.