Bug 198555 - mirror log not getting cleared causes new mirror creation to fail
mirror log not getting cleared causes new mirror creation to fail
Status: CLOSED CURRENTRELEASE
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: cmirror (Show other bugs)
4
All Linux
high Severity high
: ---
: ---
Assigned To: Jonathan Earl Brassow
Cluster QE
:
Depends On: 199185
Blocks:
  Show dependency treegraph
 
Reported: 2006-07-11 16:18 EDT by Corey Marthaler
Modified: 2010-04-27 10:56 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-04-27 10:56:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Corey Marthaler 2006-07-11 16:18:14 EDT
Description of problem:
This seems like a new version of bz 164668, but since I've only been able to
reproduce it when doing converts (not deletions) I figured this was a new bug.

# create it
[root@taft-04 ~]# lvcreate -m 1 -n convert -L 1G B
  Logical volume "convert" created

# convert it
[root@taft-02 ~]# lvconvert -m 0 /dev/B/convert
  Logical volume convert converted.
[root@taft-02 ~]# lvs
  LV      VG   Attr   LSize Origin Snap%  Move Log Copy%
  convert B    -wi-a- 1.00G
[root@taft-02 ~]# lvscan
  ACTIVE            '/dev/B/convert' [1.00 GB] inherit

# attempt to convert it back
[root@taft-04 ~]# lvconvert -m 1 /dev/B/convert
  /dev/B/convert_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log. Remove new LV and retry.
  Failed to create mirror log.
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/convert' [1.00 GB] inherit
  ACTIVE            '/dev/B/convert_mlog' [4.00 MB] inherit
[root@taft-04 ~]# lvs
  LV           VG   Attr   LSize Origin Snap%  Move Log Copy%
  convert      B    -wi-a- 1.00G
  convert_mlog B    -wi-a- 4.00M

 
Version-Release number of selected component (if applicable):
[root@taft-04 ~]# rpm -q lvm2
lvm2-2.02.06-4.0.RHEL4
[root@taft-04 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-4.0.RHEL4
[root@taft-04 ~]# rpm -q cmirror
cmirror-1.0.1-0
[root@taft-04 ~]# rpm -q cmirror-kernel
cmirror-kernel-2.6.9-10.0
Comment 1 Corey Marthaler 2006-07-11 17:14:12 EDT
Reproduced with a little more detail.

[root@taft-04 ~]# lvcreate -m 1 -n converter -L 1G B
  Logical volume "converter" created
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/converter' [1.00 GB] inherit
[root@taft-04 ~]# lvconvert -m 0 /dev/B/converter
  Logical volume converter converted.
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/converter' [1.00 GB] inherit
[root@taft-04 ~]# dmsetup status
B-converter: 0 2097152 linear
B-converter_mimage_1: 0 2097152 linear
B-converter_mimage_0: 0 2097152 linear
B-converter_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear
[root@taft-04 ~]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
B-converter      253   5 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGyImmlktc3k8O01p53RyFg0Vx5FMivlfc
B-converter_mimage_1 253   4 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGPxRhHLABtCwUYCYGFP5V7kNv1kgK4Joz
B-converter_mimage_0 253   3 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGGO3LJR74bPCtXOuPAlm1824yfWIA05mM
B-converter_mlog 253   2 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGePZxmPYUA6WxeVtIqEp7tkmCWINzKfI1
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693
[root@taft-04 ~]# lvconvert -m 1 /dev/B/converter
  /dev/B/converter_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log. Remove new LV and retry.
  Failed to create mirror log.
[root@taft-04 ~]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
B-converter      253   5 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGyImmlktc3k8O01p53RyFg0Vx5FMivlfc
B-converter_mimage_1 253   4 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGPxRhHLABtCwUYCYGFP5V7kNv1kgK4Joz
B-converter_mimage_0 253   3 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGGO3LJR74bPCtXOuPAlm1824yfWIA05mM
B-converter_mlog 253   2 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGePZxmPYUA6WxeVtIqEp7tkmCWINzKfI1
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693
[root@taft-04 ~]# dmsetup status
B-converter: 0 2097152 linear
B-converter_mimage_1: 0 2097152 linear
B-converter_mimage_0: 0 2097152 linear
B-converter_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear
Comment 2 Jonathan Earl Brassow 2006-07-11 17:16:59 EDT
fixed as a side affect of 194125 fix
Comment 3 Corey Marthaler 2006-07-13 14:42:36 EDT
I still see leftover volumes on one of the four nodes in the cluster, which
causes the devices to appear "in use". 

After competely deleting the mirror, the vg, and the pvs associated with it,
there are still dm device left over on taft-04.

[root@taft-04 tmp]# lvscan
  No volume groups found
[root@taft-04 tmp]# vgscan
  Reading all physical volumes.  This may take a while...
  No volume groups found
[root@taft-04 tmp]# pvscan
  No matching physical volumes found
[root@taft-04 tmp]# lvs
  No volume groups found
[root@taft-04 tmp]# dmsetup status
mirror_1_1758-mirror_1_17580_mimage_1: 0 285696000 linear
mirror_1_1758-mirror_1_17580_mimage_0: 0 285696000 linear
mirror_1_1758-mirror_1_17580_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear
[root@taft-04 tmp]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
mirror_1_1758-mirror_1_17580_mimage_1 253   4 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfzP018N3pDrWlDiIadrZhfgRbf3irxtkB
mirror_1_1758-mirror_1_17580_mimage_0 253   3 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfifXWKuACAdVOvmCOUFcTmRymxAho2QEg
mirror_1_1758-mirror_1_17580_mlog 253   2 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfuUaPcUKhiatuhVTEF3BbZZ2YMeOmDpzc
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693


On the other three nodes in the cluster, no dm device are left over.
[root@taft-03 tmp]# dmsetup status
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 138674176 linear
[root@taft-03 tmp]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-q1NpfwugQwN7ZTjEWCU0xZQCVfmFwW02GcSP71EoKR21ZeCkCdjweZ9QetiQOIPb
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-q1NpfwugQwN7ZTjEWCU0xZQCVfmFwW02cZTaBj3kZMhvzBWZJTY48IlMtBBDc3pd

This is with the latest rpms:
[root@taft-04 tmp]# rpm -q device-mapper
device-mapper-1.02.07-4.0.RHEL4
[root@taft-04 tmp]# rpm -q lvm2
lvm2-2.02.06-5.0.RHEL4
[root@taft-04 tmp]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-5.0.RHEL4

Comment 4 Corey Marthaler 2006-07-13 14:50:48 EDT
FWIW, it I reboot the machine or just delete the dm devices with a dmremove,
everything works fine again.
Comment 5 Corey Marthaler 2006-07-13 17:08:54 EDT
reproduced with the lvconvert example in comment #1 as well.
Comment 6 Jonathan Earl Brassow 2006-07-20 16:03:31 EDT
This won't be fixed until 199185 is fixed.
Comment 7 Jonathan Earl Brassow 2006-07-22 14:28:57 EDT
There is now a dependence chain.  198555 -> 199185 -> 199826

199826 will be fixed shortly.
199185 requires 199826 plus a fix in lvm2-cluster
I believe this bug will be resolved automatically when those are in place.
Comment 8 Corey Marthaler 2006-08-03 11:57:21 EDT
Just a note that even with all the fixes for the bugs listed in comment #7, I
still saw this issue today with the lastest code. This was while doing cmirror
creation and deletion and had nothing to do with up and down converting mirrors.
Comment 9 Corey Marthaler 2006-08-03 11:59:23 EDT
More info for comment #8:

# After mirror deletion I still saw:
[root@taft-04 ~]# dmsetup ls
mirror_1_4600-mirror_1_46000_mlog       (253, 2)

Versions:
[root@taft-04 ~]# uname -ar
Linux taft-04 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64
x86_64 GNU/Lix
[root@taft-04 ~]# rpm -q device-mapper
device-mapper-1.02.07-4.0.RHEL4
[root@taft-04 ~]# rpm -q cmirror
cmirror-1.0.1-0
[root@taft-04 ~]# rpm -q cmirror-kernel
cmirror-kernel-2.6.9-13.0
[root@taft-04 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-7.0.RHEL4
[root@taft-04 ~]# rpm -q lvm2
lvm2-2.02.06-6.0.RHEL4

Comment 10 Jonathan Earl Brassow 2006-08-07 17:07:55 EDT
were you doing any machine failures?
Comment 11 Jonathan Earl Brassow 2006-08-09 16:55:42 EDT
I've seen this happen.  For some reason all the LVM representations of the
device are cleared, but the device-mapper devices remain on one of the nodes.

The remove (or convert) is the real culprit, but a future create (up convert)
will run into the old log device and fail in the aforemention way.

Working on finding the cause.
Comment 12 Jonathan Earl Brassow 2006-09-08 15:41:13 EDT
This could be caused by multiple issues:

1) inability for cluster mirrors to move the server during suspension
2) LVM not activating the lv before removal (causing the dm device to be ignored)

#1 has been resolved
#2 has been fixed in lvm2-cluster and is waiting to go into lvm2
Comment 13 Corey Marthaler 2006-11-07 18:27:36 EST
fix verified in lvm2-cluster-2.02.13-1.
Comment 15 Alasdair Kergon 2010-04-27 10:56:00 EDT
Assuming this VERIFIED fix got released.  Closing.
Reopen if it's not yet resolved.

Note You need to log in before you can comment on or make changes to this bug.