Bug 214529 - vgremove reports LV exists after all lvs have been deleted
vgremove reports LV exists after all lvs have been deleted
Status: CLOSED CURRENTRELEASE
Product: Red Hat Cluster Suite
Classification: Red Hat
Component: lvm2-cluster (Show other bugs)
4
All Linux
medium Severity medium
: ---
: ---
Assigned To: Alasdair Kergon
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2006-11-07 18:44 EST by Corey Marthaler
Modified: 2010-01-11 23:05 EST (History)
4 users (show)

See Also:
Fixed In Version: RC
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-02-07 21:22:58 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
metadata (187.46 KB, application/octet-stream)
2006-11-08 12:31 EST, Corey Marthaler
no flags Details
Fossil records (4.02 KB, application/x-compressed-tar)
2006-11-14 19:22 EST, Jonathan Earl Brassow
no flags Details
patch (1007 bytes, text/x-patch)
2006-11-20 17:53 EST, Jonathan Earl Brassow
no flags Details
patch (1007 bytes, patch)
2006-11-20 17:55 EST, Jonathan Earl Brassow
no flags Details | Diff
Good Patch (1.64 KB, patch)
2006-11-20 18:43 EST, Jonathan Earl Brassow
no flags Details | Diff

  None (edit)
Description Corey Marthaler 2006-11-07 18:44:15 EST
Description of problem:
I was unable to remove a vg due to there "still being a LV present".
This may just be a single lvm bug, but the only time I've ever seen this was
with clvmd. Plus it seems like the cluster aspect may be what causing this
phantom LV.


[root@link-02 ~]# pvscan
  PV /dev/sda1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdb1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdc1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]
[root@link-02 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "corey" using metadata type lvm2
[root@link-02 ~]# lvscan
[root@link-02 ~]# lvs -a -o +devices
[root@link-02 ~]# dmsetup ls
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-02 ~]# vgremove corey
  Volume group "corey" still contains 1 logical volume(s)


[root@link-04 ~]# pvscan
  PV /dev/sda1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdb1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdc1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]
[root@link-04 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "corey" using metadata type lvm2
[root@link-04 ~]# lvscan
[root@link-04 ~]# lvs -a -o +devices
[root@link-04 ~]# dmsetup ls
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-04 ~]# vgremove corey
  Volume group "corey" still contains 1 logical volume(s)


root@link-07 ~]# pvscan
  PV /dev/sda1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdb1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdc1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]
[root@link-07 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "corey" using metadata type lvm2
[root@link-07 ~]# lvscan
[root@link-07 ~]# lvs -a -o +devices
[root@link-07 ~]# dmsetup ls
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-07 ~]# vgremove corey
  Volume group "corey" still contains 1 logical volume(s)


[root@link-08 ~]# pvscan
  PV /dev/sda1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdb1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdc1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG corey   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]
[root@link-08 ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "corey" using metadata type lvm2
[root@link-08 ~]# lvscan
[root@link-08 ~]# lvs -a -o +devices
[root@link-08 ~]# dmsetup ls
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-08 ~]# vgremove corey
  Volume group "corey" still contains 1 logical volume(s)


Version-Release number of selected component (if applicable):
[root@link-02 ~]# rpm -qa | grep lvm2
lvm2-cluster-2.02.13-1
lvm2-cluster-debuginfo-2.02.06-7.0.RHEL4
lvm2-2.02.13-1
Comment 1 Christine Caulfield 2006-11-08 05:06:20 EST
It would be intersting to know what the other cluster members thought the VG
state was - and what the lvm metadata looked like.
Comment 2 Corey Marthaler 2006-11-08 12:31:22 EST
Created attachment 140670 [details]
metadata
Comment 3 Christine Caulfield 2006-11-10 09:19:45 EST
hmm, though the metadata looks a bit garbled, there seem to be quite a few
mirror components listed in there. brassow may be interested it in that.
Comment 4 Christine Caulfield 2006-11-13 08:40:39 EST
Assign to Jon as it looks (to me) like there are mirror components being left
lying around, these would be hidden from lvs command (though not lvs -a).

Send it back if that's not true :)
Comment 5 Jonathan Earl Brassow 2006-11-13 16:59:35 EST
can you reproduce this?

and what happens when you do a 'lvremove -ff corey' before trying the 'vgremove
corey'.  (Yes, I understand that whatever lv is blocking is not being printed.)  
Comment 6 Jonathan Earl Brassow 2006-11-14 16:12:14 EST
Looks like we were in the process of down converting/ removing...  were did this
"mirror3_mimage_0" come from...
Looking at the attachment, seqno = 558 is the last one issued.

555:
"mirror"
"mirror_mlog
"mirror_mimage_1"
"mirror_mimage_0"
"mirror3_mimage_0" - still there...

556:
"mirror" - now linear
"mirror_mlog
"mirror_mimage_1"
"mirror_mimage_0" - now with zero seg count
"mirror3_mimage_0" - still there...

557:
"mirror" - linear
"mirror3_mimage_0" still there...

558:
"mirror3_mimage_0" still there...
Comment 7 Jonathan Earl Brassow 2006-11-14 16:14:31 EST
the history of were "mirror3_mimage_0" came from has long since been lost by the
circular buffer.  (Hundreds of changes since it was created.)

Recreatable?
Comment 8 Alasdair Kergon 2006-11-14 16:16:17 EST
(there was a very old bug that used to leave mirror images behind - also could
be udev related)
Comment 9 Jonathan Earl Brassow 2006-11-14 18:47:09 EST
I have not gotten LVM to run across this problem on its own...  However, if I
inject an LV with the same characteristics of Corey's "mirror3_mimage_0" (i.e.
an LV with 0 segments), I get the same results.

The work-around that I suggested above solves the issue - 'lvremove -ff <vg name>'.

How you ever got into this state is another question.  Reproduce, and we'll
talk.  :)

[root@neo-04 ~]# lvs
  LV         VG           Attr   LSize   Origin Snap%  Move Log Copy%  Devices
  LogVolKS00 VolGroupKS00 -wi-ao  36.62G                               /dev/hda2(0)
  LogVolKS01 VolGroupKS00 -wi-ao 512.00M                              
/dev/hda2(1172)
[root@neo-04 ~]# dmsetup ls
VolGroupKS00-LogVolKS01 (253, 1)
VolGroupKS00-LogVolKS00 (253, 0)
[root@neo-04 ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree   VG Tags
  VolGroupKS00   1   2   0 wz--n-  37.16G  32.00M neo-04.lab.msp.redhat.com
  vg             7   1   0 wz--nc 120.01G 120.01G
[root@neo-04 ~]# vgremove vg
  Volume group "vg" still contains 1 logical volume(s)
[root@neo-04 ~]# lvremove -ff vg
  Logical volume "brassow" successfully removed
[root@neo-04 ~]# vgremove vg
  Volume group "vg" successfully removed
Comment 10 Jonathan Earl Brassow 2006-11-14 18:52:34 EST
In response to comment #8...

This is not the old bug that left mirror images behind.  That was due to
dmeventd waiting on a device and therefore not allowing device-mapper to pull it
out.

This bug has to do with conversion.  If you look at the stages outlined in
comment #6, you can see that one of the images goes to 0 segments before being
removed on the conversion.  If it is not removed, this bug will occur.  I don't
know if this bug affects only the LV with 0 segments or if it could affect them
all - Corey's trace didn't go back far enough.
Comment 11 Jonathan Earl Brassow 2006-11-14 19:05:29 EST
hmmm, found corey's archives on link-02  :)
Comment 12 Jonathan Earl Brassow 2006-11-14 19:22:35 EST
Created attachment 141202 [details]
Fossil records

all the archives from when mirror3 existed to when it got screwed over. 
(Pulled from link-02...  I don't know if the other machine(s) was also doing
operations.)
Comment 13 Alasdair Kergon 2006-11-15 08:59:11 EST
but could it still have been udev holding the device preventing lvm2 removing it?
Comment 14 Jonathan Earl Brassow 2006-11-20 17:53:18 EST
Created attachment 141703 [details]
patch
Comment 15 Jonathan Earl Brassow 2006-11-20 17:55:38 EST
Created attachment 141704 [details]
patch

We need to remove the phantom LV (<lvname>_mimage_0) that has no segments
before writing out the metadata.
Comment 16 Jonathan Earl Brassow 2006-11-20 18:43:05 EST
Created attachment 141710 [details]
Good Patch

We no longer write out an lv that has 0 segments in the metadata.  We instead
replace the segment with the "error" segtype.
Comment 17 Jonathan Earl Brassow 2006-11-20 18:45:39 EST
assigning to agk to review, add to repo, and mark as POST/MODIFIED.
Comment 18 RHEL Product and Program Management 2006-11-30 13:30:47 EST
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux major release.  Product Management has requested further
review of this request by Red Hat Engineering, for potential inclusion in a Red
Hat Enterprise Linux Major release.  This request is not yet committed for
inclusion.
Comment 19 Corey Marthaler 2007-01-24 16:07:12 EST
This bug has not been seen with the lastest lvm2 builds, marking verified.
Comment 20 RHEL Product and Program Management 2007-02-07 21:22:58 EST
A package has been built which should help the problem described in 
this bug report. This report is therefore being closed with a resolution 
of CURRENTRELEASE. You may reopen this bug report if the solution does 
not work for you.

Note You need to log in before you can comment on or make changes to this bug.