Bug 198555

Summary:	mirror log not getting cleared causes new mirror creation to fail
Product:	[Retired] Red Hat Cluster Suite	Reporter:	Corey Marthaler <cmarthal>
Component:	cmirror	Assignee:	Jonathan Earl Brassow <jbrassow>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Cluster QE <mspqa-list>
Severity:	high	Docs Contact:
Priority:	high
Version:	4	CC:	agk, dwysocha, mbroz
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2010-04-27 14:56:00 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	199185
Bug Blocks:

Description Corey Marthaler 2006-07-11 20:18:14 UTC

Description of problem:
This seems like a new version of bz 164668, but since I've only been able to
reproduce it when doing converts (not deletions) I figured this was a new bug.

# create it
[root@taft-04 ~]# lvcreate -m 1 -n convert -L 1G B
  Logical volume "convert" created

# convert it
[root@taft-02 ~]# lvconvert -m 0 /dev/B/convert
  Logical volume convert converted.
[root@taft-02 ~]# lvs
  LV      VG   Attr   LSize Origin Snap%  Move Log Copy%
  convert B    -wi-a- 1.00G
[root@taft-02 ~]# lvscan
  ACTIVE            '/dev/B/convert' [1.00 GB] inherit

# attempt to convert it back
[root@taft-04 ~]# lvconvert -m 1 /dev/B/convert
  /dev/B/convert_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log. Remove new LV and retry.
  Failed to create mirror log.
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/convert' [1.00 GB] inherit
  ACTIVE            '/dev/B/convert_mlog' [4.00 MB] inherit
[root@taft-04 ~]# lvs
  LV           VG   Attr   LSize Origin Snap%  Move Log Copy%
  convert      B    -wi-a- 1.00G
  convert_mlog B    -wi-a- 4.00M

 
Version-Release number of selected component (if applicable):
[root@taft-04 ~]# rpm -q lvm2
lvm2-2.02.06-4.0.RHEL4
[root@taft-04 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-4.0.RHEL4
[root@taft-04 ~]# rpm -q cmirror
cmirror-1.0.1-0
[root@taft-04 ~]# rpm -q cmirror-kernel
cmirror-kernel-2.6.9-10.0

Comment 1 Corey Marthaler 2006-07-11 21:14:12 UTC

Reproduced with a little more detail.

[root@taft-04 ~]# lvcreate -m 1 -n converter -L 1G B
  Logical volume "converter" created
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/converter' [1.00 GB] inherit
[root@taft-04 ~]# lvconvert -m 0 /dev/B/converter
  Logical volume converter converted.
[root@taft-04 ~]# lvscan
  ACTIVE            '/dev/B/converter' [1.00 GB] inherit
[root@taft-04 ~]# dmsetup status
B-converter: 0 2097152 linear
B-converter_mimage_1: 0 2097152 linear
B-converter_mimage_0: 0 2097152 linear
B-converter_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear
[root@taft-04 ~]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
B-converter      253   5 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGyImmlktc3k8O01p53RyFg0Vx5FMivlfc
B-converter_mimage_1 253   4 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGPxRhHLABtCwUYCYGFP5V7kNv1kgK4Joz
B-converter_mimage_0 253   3 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGGO3LJR74bPCtXOuPAlm1824yfWIA05mM
B-converter_mlog 253   2 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGePZxmPYUA6WxeVtIqEp7tkmCWINzKfI1
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693
[root@taft-04 ~]# lvconvert -m 1 /dev/B/converter
  /dev/B/converter_mlog: not found: device not cleared
  Aborting. Failed to wipe mirror log. Remove new LV and retry.
  Failed to create mirror log.
[root@taft-04 ~]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
B-converter      253   5 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGyImmlktc3k8O01p53RyFg0Vx5FMivlfc
B-converter_mimage_1 253   4 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGPxRhHLABtCwUYCYGFP5V7kNv1kgK4Joz
B-converter_mimage_0 253   3 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGGO3LJR74bPCtXOuPAlm1824yfWIA05mM
B-converter_mlog 253   2 L--w    0    1      0
LVM-06KWBzrFMwUp36z4d0dSg4qpBbe1LyFGePZxmPYUA6WxeVtIqEp7tkmCWINzKfI1
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693
[root@taft-04 ~]# dmsetup status
B-converter: 0 2097152 linear
B-converter_mimage_1: 0 2097152 linear
B-converter_mimage_0: 0 2097152 linear
B-converter_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear

Comment 2 Jonathan Earl Brassow 2006-07-11 21:16:59 UTC

fixed as a side affect of 194125 fix

Comment 3 Corey Marthaler 2006-07-13 18:42:36 UTC

I still see leftover volumes on one of the four nodes in the cluster, which
causes the devices to appear "in use". 

After competely deleting the mirror, the vg, and the pvs associated with it,
there are still dm device left over on taft-04.

[root@taft-04 tmp]# lvscan
  No volume groups found
[root@taft-04 tmp]# vgscan
  Reading all physical volumes.  This may take a while...
  No volume groups found
[root@taft-04 tmp]# pvscan
  No matching physical volumes found
[root@taft-04 tmp]# lvs
  No volume groups found
[root@taft-04 tmp]# dmsetup status
mirror_1_1758-mirror_1_17580_mimage_1: 0 285696000 linear
mirror_1_1758-mirror_1_17580_mimage_0: 0 285696000 linear
mirror_1_1758-mirror_1_17580_mlog: 0 8192 linear
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 40960000 linear
[root@taft-04 tmp]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
mirror_1_1758-mirror_1_17580_mimage_1 253   4 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfzP018N3pDrWlDiIadrZhfgRbf3irxtkB
mirror_1_1758-mirror_1_17580_mimage_0 253   3 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfifXWKuACAdVOvmCOUFcTmRymxAho2QEg
mirror_1_1758-mirror_1_17580_mlog 253   2 L--w    0    1      0
LVM-q729ZhdqDQyNi1t5NGF3F1hlcpKcB0BfuUaPcUKhiatuhVTEF3BbZZ2YMeOmDpzc
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5UsN964on4p3rGGX2nMO3T4sYHE3TnkhP
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-zUByyTfjW0kZbVav84sw7MJSW2JYnqw5ic586KqB9N623SZmXmWa54kFJ3nZM693


On the other three nodes in the cluster, no dm device are left over.
[root@taft-03 tmp]# dmsetup status
VolGroup00-LogVol01: 0 4063232 linear
VolGroup00-LogVol00: 0 138674176 linear
[root@taft-03 tmp]# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID
VolGroup00-LogVol01 253   1 L--w    1    1      0
LVM-q1NpfwugQwN7ZTjEWCU0xZQCVfmFwW02GcSP71EoKR21ZeCkCdjweZ9QetiQOIPb
VolGroup00-LogVol00 253   0 L--w    1    1      0
LVM-q1NpfwugQwN7ZTjEWCU0xZQCVfmFwW02cZTaBj3kZMhvzBWZJTY48IlMtBBDc3pd

This is with the latest rpms:
[root@taft-04 tmp]# rpm -q device-mapper
device-mapper-1.02.07-4.0.RHEL4
[root@taft-04 tmp]# rpm -q lvm2
lvm2-2.02.06-5.0.RHEL4
[root@taft-04 tmp]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-5.0.RHEL4

Comment 4 Corey Marthaler 2006-07-13 18:50:48 UTC

FWIW, it I reboot the machine or just delete the dm devices with a dmremove,
everything works fine again.

Comment 5 Corey Marthaler 2006-07-13 21:08:54 UTC

reproduced with the lvconvert example in comment #1 as well.

Comment 6 Jonathan Earl Brassow 2006-07-20 20:03:31 UTC

This won't be fixed until 199185 is fixed.

Comment 7 Jonathan Earl Brassow 2006-07-22 18:28:57 UTC

There is now a dependence chain.  198555 -> 199185 -> 199826

199826 will be fixed shortly.
199185 requires 199826 plus a fix in lvm2-cluster
I believe this bug will be resolved automatically when those are in place.

Comment 8 Corey Marthaler 2006-08-03 15:57:21 UTC

Just a note that even with all the fixes for the bugs listed in comment #7, I
still saw this issue today with the lastest code. This was while doing cmirror
creation and deletion and had nothing to do with up and down converting mirrors.

Comment 9 Corey Marthaler 2006-08-03 15:59:23 UTC

More info for comment #8:

# After mirror deletion I still saw:
[root@taft-04 ~]# dmsetup ls
mirror_1_4600-mirror_1_46000_mlog       (253, 2)

Versions:
[root@taft-04 ~]# uname -ar
Linux taft-04 2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:32:02 EDT 2006 x86_64 x86_64
x86_64 GNU/Lix
[root@taft-04 ~]# rpm -q device-mapper
device-mapper-1.02.07-4.0.RHEL4
[root@taft-04 ~]# rpm -q cmirror
cmirror-1.0.1-0
[root@taft-04 ~]# rpm -q cmirror-kernel
cmirror-kernel-2.6.9-13.0
[root@taft-04 ~]# rpm -q lvm2-cluster
lvm2-cluster-2.02.06-7.0.RHEL4
[root@taft-04 ~]# rpm -q lvm2
lvm2-2.02.06-6.0.RHEL4

Comment 10 Jonathan Earl Brassow 2006-08-07 21:07:55 UTC

were you doing any machine failures?

Comment 11 Jonathan Earl Brassow 2006-08-09 20:55:42 UTC

I've seen this happen.  For some reason all the LVM representations of the
device are cleared, but the device-mapper devices remain on one of the nodes.

The remove (or convert) is the real culprit, but a future create (up convert)
will run into the old log device and fail in the aforemention way.

Working on finding the cause.

Comment 12 Jonathan Earl Brassow 2006-09-08 19:41:13 UTC

This could be caused by multiple issues:

1) inability for cluster mirrors to move the server during suspension
2) LVM not activating the lv before removal (causing the dm device to be ignored)

#1 has been resolved
#2 has been fixed in lvm2-cluster and is waiting to go into lvm2

Comment 13 Corey Marthaler 2006-11-07 23:27:36 UTC

fix verified in lvm2-cluster-2.02.13-1.

Comment 15 Alasdair Kergon 2010-04-27 14:56:00 UTC

Assuming this VERIFIED fix got released.  Closing.
Reopen if it's not yet resolved.