Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 744002

Summary: cmirror conversion failures due to cluster locking errors
Product: Red Hat Enterprise Linux 6 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED ERRATA QA Contact: Cluster QE <mspqa-list>
Severity: high Docs Contact:
Priority: high    
Version: 6.2CC: agk, dwysocha, heinzm, jbrassow, mbroz, prajnoha, prockai, thornber, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: lvm2-2.02.87-5.el6 Doc Type: Bug Fix
Doc Text:
Do not document.
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 17:03:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 747120    

Description Corey Marthaler 2011-10-06 18:02:39 UTC
Description of problem:
This may be related to bug 733114.

 TEST CASE=58
  base legs 1; convert to 0 legs;
  base log disk; convert to core log;
  active 1; sync=1; pvs=none
 ---------------------------------------------
 creating a base mirror on taft-01
 Waiting until all mirrors become fully syncd...
    0/1 mirror(s) are fully synced: ( 22.36% )
    0/1 mirror(s) are fully synced: ( 47.43% )
    0/1 mirror(s) are fully synced: ( 68.07% )
    0/1 mirror(s) are fully synced: ( 94.29% )
    1/1 mirror(s) are fully synced: ( 100.00% )
 Converting from 1 leg(s) disk log; to 0 leg(s) core log on taft-01
 lvconvert --mirrorlog core -m 0 centipede/centi_base
   Error locking on node taft-04: LV centipede/centi_base_mimage_1 in use: not deactivating
   Error locking on node taft-03: LV centipede/centi_base_mimage_1 in use: not deactivating
   Error locking on node taft-02: LV centipede/centi_base_mimage_1 in use: not deactivating
 convert case failed on taft-01


TAFT-01:
[root@taft-01 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi--- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-02:
[root@taft-02 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                    
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-03:
[root@taft-03 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-04:
[root@taft-04 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    


Version-Release number of selected component (if applicable):
2.6.32-203.el6.x86_64

lvm2-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
lvm2-libs-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
lvm2-cluster-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-libs-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-event-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-event-libs-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
cmirror-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011


How reproducible:
Often

Comment 1 Corey Marthaler 2011-10-06 20:39:42 UTC
A couple different tests/testcases are hitting this:

 SCENARIO - [looping_mirror_to_linear_converts]
 Create a mirror and then down and up convert it 20 times
 taft-01: lvcreate -m 1 -n mirror_2_linear -L 300M --nosync mirror_sanity
   WARNING: New mirror won't be synchronised. Don't read what you didn't write!
 1: down convert to linear on taft-04;
   Error locking on node taft-02: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
   Error locking on node taft-03: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
   Error locking on node taft-04: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
 down converting the mirror failed

Comment 2 Jonathan Earl Brassow 2011-10-10 23:05:02 UTC
First test (centipede) passed w/o failure:

---------------------------------------------
TEST CASE=72
 base legs 2; convert to 2 legs;
 base log disk; convert to disk log;
 active 1; sync=1; pvs=none
---------------------------------------------
creating a base mirror on bp-01
Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 16.50% )
   0/1 mirror(s) are fully synced: ( 32.79% )
   0/1 mirror(s) are fully synced: ( 48.93% )
   0/1 mirror(s) are fully synced: ( 65.29% )
   0/1 mirror(s) are fully synced: ( 81.64% )
   0/1 mirror(s) are fully synced: ( 97.79% )
   1/1 mirror(s) are fully synced: ( 100.00% )
Attempting convert from 2 leg(s) disk log to 2 leg(s) disk log on bp-02
(THIS IS A NO-OP, but should still be attempted)
lvconvert --mirrorlog disk -m 2 centipede/centi_base

Removing base volume on bp-03

Comment 3 Jonathan Earl Brassow 2011-10-11 13:30:01 UTC
mirror_sanity ran all night...

============================================================
Iteration 77 of 10000 started at Tue Oct 11 08:12:45 CDT 2011
============================================================
SCENARIO - [looping_mirror_to_linear_converts]
Create a mirror and then down and up convert it 20 times
bp-01: lvcreate -m 1 -n mirror_2_linear -L 300M --nosync mirror_sanity
  WARNING: New mirror won't be synchronised. Don't read what you didn't write!
1: down convert to linear on bp-01; up convert on mirror bp-01
2: down convert to linear on bp-04; up convert on mirror bp-02
3: down convert to linear on bp-03; up convert on mirror bp-01
4: down convert to linear on bp-03; up convert on mirror bp-04
5: down convert to linear on bp-04; up convert on mirror bp-03
6: down convert to linear on bp-03; up convert on mirror bp-03
7: down convert to linear on bp-02; up convert on mirror bp-04
8: down convert to linear on bp-01; up convert on mirror bp-01
9: down convert to linear on bp-01; up convert on mirror bp-04
10: down convert to linear on bp-01; up convert on mirror bp-02
11: down convert to linear on bp-04; up convert on mirror bp-04
12: down convert to linear on bp-02; up convert on mirror bp-04
13: down convert to linear on bp-03; up convert on mirror bp-01
14: down convert to linear on bp-03; up convert on mirror bp-04
15: down convert to linear on bp-02; up convert on mirror bp-03
16: down convert to linear on bp-04; up convert on mirror bp-01
17: down convert to linear on bp-04; up convert on mirror bp-03
18: down convert to linear on bp-02; up convert on mirror bp-04
19: down convert to linear on bp-02; up convert on mirror bp-02
20: down convert to linear on bp-03; up convert on mirror bp-03
Deactivating mirror mirror_2_linear... and removing

This bug seems to have the same root cause as 733114 - the same commits will work here.

Comment 7 Corey Marthaler 2011-10-13 15:12:56 UTC
The cmirror conversion regression test now passes. Marking verified.

2.6.32-207.el6.x86_64

lvm2-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
lvm2-libs-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
lvm2-cluster-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-libs-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-event-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-event-libs-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
cmirror-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011

Comment 8 Peter Rajnoha 2011-10-27 11:12:07 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Do not document.

Comment 9 errata-xmlrpc 2011-12-06 17:03:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1522.html