Bug 744002 - cmirror conversion failures due to cluster locking errors
Summary: cmirror conversion failures due to cluster locking errors
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Jonathan Earl Brassow
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks: 747120
TreeView+ depends on / blocked
 
Reported: 2011-10-06 18:02 UTC by Corey Marthaler
Modified: 2011-12-06 17:03 UTC (History)
9 users (show)

Fixed In Version: lvm2-2.02.87-5.el6
Doc Type: Bug Fix
Doc Text:
Do not document.
Clone Of:
Environment:
Last Closed: 2011-12-06 17:03:44 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2011:1522 0 normal SHIPPED_LIVE lvm2 bug fix and enhancement update 2011-12-06 00:50:10 UTC

Description Corey Marthaler 2011-10-06 18:02:39 UTC
Description of problem:
This may be related to bug 733114.

 TEST CASE=58
  base legs 1; convert to 0 legs;
  base log disk; convert to core log;
  active 1; sync=1; pvs=none
 ---------------------------------------------
 creating a base mirror on taft-01
 Waiting until all mirrors become fully syncd...
    0/1 mirror(s) are fully synced: ( 22.36% )
    0/1 mirror(s) are fully synced: ( 47.43% )
    0/1 mirror(s) are fully synced: ( 68.07% )
    0/1 mirror(s) are fully synced: ( 94.29% )
    1/1 mirror(s) are fully synced: ( 100.00% )
 Converting from 1 leg(s) disk log; to 0 leg(s) core log on taft-01
 lvconvert --mirrorlog core -m 0 centipede/centi_base
   Error locking on node taft-04: LV centipede/centi_base_mimage_1 in use: not deactivating
   Error locking on node taft-03: LV centipede/centi_base_mimage_1 in use: not deactivating
   Error locking on node taft-02: LV centipede/centi_base_mimage_1 in use: not deactivating
 convert case failed on taft-01


TAFT-01:
[root@taft-01 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi--- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-02:
[root@taft-02 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                    
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-03:
[root@taft-03 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    

TAFT-04:
[root@taft-04 ~]# lvs -a -o +devices
  LV                  VG        Attr   LSize   Devices         
  centi_base          centipede -wi-a- 700.00m /dev/sdh2(0)    
  centi_base_mimage_0 centipede vwi-a- 700.00m                 
  centi_base_mimage_1 centipede -wi-a- 700.00m /dev/sdh1(0)    
  centi_base_mlog     centipede -wi-s-   4.00m /dev/sdd2(0)    


Version-Release number of selected component (if applicable):
2.6.32-203.el6.x86_64

lvm2-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
lvm2-libs-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
lvm2-cluster-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-libs-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-event-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
device-mapper-event-libs-1.02.66-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011
cmirror-2.02.87-3.el6    BUILT: Wed Sep 21 09:54:55 CDT 2011


How reproducible:
Often

Comment 1 Corey Marthaler 2011-10-06 20:39:42 UTC
A couple different tests/testcases are hitting this:

 SCENARIO - [looping_mirror_to_linear_converts]
 Create a mirror and then down and up convert it 20 times
 taft-01: lvcreate -m 1 -n mirror_2_linear -L 300M --nosync mirror_sanity
   WARNING: New mirror won't be synchronised. Don't read what you didn't write!
 1: down convert to linear on taft-04;
   Error locking on node taft-02: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
   Error locking on node taft-03: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
   Error locking on node taft-04: LV mirror_sanity/mirror_2_linear_mimage_1 in use: not deactivating
 down converting the mirror failed

Comment 2 Jonathan Earl Brassow 2011-10-10 23:05:02 UTC
First test (centipede) passed w/o failure:

---------------------------------------------
TEST CASE=72
 base legs 2; convert to 2 legs;
 base log disk; convert to disk log;
 active 1; sync=1; pvs=none
---------------------------------------------
creating a base mirror on bp-01
Waiting until all mirrors become fully syncd...
   0/1 mirror(s) are fully synced: ( 16.50% )
   0/1 mirror(s) are fully synced: ( 32.79% )
   0/1 mirror(s) are fully synced: ( 48.93% )
   0/1 mirror(s) are fully synced: ( 65.29% )
   0/1 mirror(s) are fully synced: ( 81.64% )
   0/1 mirror(s) are fully synced: ( 97.79% )
   1/1 mirror(s) are fully synced: ( 100.00% )
Attempting convert from 2 leg(s) disk log to 2 leg(s) disk log on bp-02
(THIS IS A NO-OP, but should still be attempted)
lvconvert --mirrorlog disk -m 2 centipede/centi_base

Removing base volume on bp-03

Comment 3 Jonathan Earl Brassow 2011-10-11 13:30:01 UTC
mirror_sanity ran all night...

============================================================
Iteration 77 of 10000 started at Tue Oct 11 08:12:45 CDT 2011
============================================================
SCENARIO - [looping_mirror_to_linear_converts]
Create a mirror and then down and up convert it 20 times
bp-01: lvcreate -m 1 -n mirror_2_linear -L 300M --nosync mirror_sanity
  WARNING: New mirror won't be synchronised. Don't read what you didn't write!
1: down convert to linear on bp-01; up convert on mirror bp-01
2: down convert to linear on bp-04; up convert on mirror bp-02
3: down convert to linear on bp-03; up convert on mirror bp-01
4: down convert to linear on bp-03; up convert on mirror bp-04
5: down convert to linear on bp-04; up convert on mirror bp-03
6: down convert to linear on bp-03; up convert on mirror bp-03
7: down convert to linear on bp-02; up convert on mirror bp-04
8: down convert to linear on bp-01; up convert on mirror bp-01
9: down convert to linear on bp-01; up convert on mirror bp-04
10: down convert to linear on bp-01; up convert on mirror bp-02
11: down convert to linear on bp-04; up convert on mirror bp-04
12: down convert to linear on bp-02; up convert on mirror bp-04
13: down convert to linear on bp-03; up convert on mirror bp-01
14: down convert to linear on bp-03; up convert on mirror bp-04
15: down convert to linear on bp-02; up convert on mirror bp-03
16: down convert to linear on bp-04; up convert on mirror bp-01
17: down convert to linear on bp-04; up convert on mirror bp-03
18: down convert to linear on bp-02; up convert on mirror bp-04
19: down convert to linear on bp-02; up convert on mirror bp-02
20: down convert to linear on bp-03; up convert on mirror bp-03
Deactivating mirror mirror_2_linear... and removing

This bug seems to have the same root cause as 733114 - the same commits will work here.

Comment 7 Corey Marthaler 2011-10-13 15:12:56 UTC
The cmirror conversion regression test now passes. Marking verified.

2.6.32-207.el6.x86_64

lvm2-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
lvm2-libs-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
lvm2-cluster-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
udev-147-2.40.el6    BUILT: Fri Sep 23 07:51:13 CDT 2011
device-mapper-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-libs-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-event-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
device-mapper-event-libs-1.02.66-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011
cmirror-2.02.87-5.el6    BUILT: Wed Oct 12 10:47:46 CDT 2011

Comment 8 Peter Rajnoha 2011-10-27 11:12:07 UTC
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Do not document.

Comment 9 errata-xmlrpc 2011-12-06 17:03:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1522.html


Note You need to log in before you can comment on or make changes to this bug.