Bug 677739

Summary: removing final exclusive cmirror snapshot, results in clvmd deadlock
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED ERRATA QA Contact: Corey Marthaler <cmarthal>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: agk, dwysocha, heinzm, jbrassow, mbroz, prajnoha, prockai, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.83-3.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 14:27:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 527186    

Description Corey Marthaler 2011-02-15 17:45:17 UTC 
Description of problem:

This issue is already described in:
https://bugzilla.redhat.com/show_bug.cgi?id=676909#c5.

[root@taft-01 ~]# lvs -a -o +devices
  LV                        VG        Attr   LSize   Log                 Copy%  Devices
  cmirror-origin            taft      mwi-a- 100.00m cmirror-origin_mlog 100.00 cmirror-origin_mimage_0(0),cmirror-origin_mimage_1(0)
  [cmirror-origin_mimage_0] taft      iwi-ao 100.00m                            /dev/sdb1(0)
  [cmirror-origin_mimage_1] taft      iwi-ao 100.00m                            /dev/sdc1(0)
  [cmirror-origin_mlog]     taft      lwi-ao   4.00m                            /dev/sdh1(0)

[root@taft-01 ~]# lvchange -an taft/cmirror-origin
[root@taft-01 ~]# lvchange -aye taft/cmirror-origin

[root@taft-01 ~]# lvcreate -s taft/cmirror-origin -L 50M -n snap1
  Rounding up size to full physical extent 52.00 MiB
  Logical volume "snap1" created
[root@taft-01 ~]# lvcreate -s taft/cmirror-origin -L 50M -n snap2
  Rounding up size to full physical extent 52.00 MiB
  Logical volume "snap2" created
[root@taft-01 ~]# lvcreate -s taft/cmirror-origin -L 50M -n snap3
  Rounding up size to full physical extent 52.00 MiB
  Logical volume "snap3" created

[root@taft-01 ~]# lvs
  LV             VG        Attr   LSize   Origin         Snap%  Log                 Copy%  
  cmirror-origin taft      owi-a- 100.00m                       cmirror-origin_mlog 100.00
  snap1          taft      swi-a-  52.00m cmirror-origin   0.00
  snap2          taft      swi-a-  52.00m cmirror-origin   0.00
  snap3          taft      swi-a-  52.00m cmirror-origin   0.00

[root@taft-01 ~]# lvremove taft/snap1
Do you really want to remove active clustered logical volume snap1? [y/n]: y
  Logical volume "snap1" successfully removed
[root@taft-01 ~]# lvremove taft/snap2
Do you really want to remove active clustered logical volume snap2? [y/n]: y
  Logical volume "snap2" successfully removed
[root@taft-01 ~]# lvremove taft/snap3
Do you really want to remove active clustered logical volume snap3? [y/n]: y
  Error locking on node taft-01: Command timed out
  Failed to resume cmirror-origin.
  Error locking on node taft-01: Command timed out

[DEADLOCK]


Version-Release number of selected component (if applicable):
2.6.32-94.el6.x86_64

lvm2-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
lvm2-libs-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
lvm2-cluster-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
udev-147-2.31.el6    BUILT: Wed Jan 26 05:39:15 CST 2011
device-mapper-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-libs-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-event-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
device-mapper-event-libs-1.02.62-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011
cmirror-2.02.83-2.el6    BUILT: Tue Feb  8 10:10:57 CST 2011


How reproducible:
everytime
Comment 1 Corey Marthaler 2011-02-15 17:58:01 UTC 
I verified that this only happens with exclusive cluster mirror snaps, not exclusive cluster linears.
Comment 2 Jonathan Earl Brassow 2011-02-15 18:41:32 UTC 
Must either find solution or disallow snaps of mirrors in a cluster...
Comment 3 Jonathan Earl Brassow 2011-02-16 23:16:23 UTC 
The creation of the snapshot preserves the exclusive lock on the origin LV, but it causes a reload of the table that uses the cluster log.  So, we go from single machine mirror to cluster mirror on one node...
Comment 4 Jonathan Earl Brassow 2011-02-17 21:04:05 UTC 
Patch posted to lvm-devel... waiting for feedback before checking-in
Comment 5 Jonathan Earl Brassow 2011-02-18 00:36:57 UTC 
upstream in version 2.02.85
Comment 7 Corey Marthaler 2011-03-18 20:09:16 UTC 
This test case now works with the latest rpms.

2.6.32-94.el6.x86_64

lvm2-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-libs-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
lvm2-cluster-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
udev-147-2.31.el6    BUILT: Wed Jan 26 05:39:15 CST 2011
device-mapper-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
device-mapper-event-libs-1.02.62-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011
cmirror-2.02.83-3.el6    BUILT: Fri Mar 18 09:31:10 CDT 2011



SCENARIO - [snaphot_exclusive_mirror]
Snapshot an exclusively activated mirror
taft-03: lvcreate -m 1 -n exclusive_origin -L 100M mirror_sanity
Deactivate and then exclusively activate mirror
Taking multiple snapshots of exclusive mirror
        1 2 3 4 5 
Removing snapshots of exclusive mirror
        1 2 3 4 5 
Deactivating mirror exclusive_origin... and removing
Comment 8 errata-xmlrpc 2011-05-19 14:27:22 UTC 
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0772.html