Bug 207470 - Need ability to handle duplicate VG names for Xen or kvm
Summary: Need ability to handle duplicate VG names for Xen or kvm
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: lvm2
Version: 19
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: LVM and device-mapper development team
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-09-21 10:04 UTC by Stephen Tweedie
Modified: 2015-02-18 11:56 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-02-18 11:56:26 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Tool to edit PV fields, including VG name (2.61 KB, text/plain)
2010-08-17 21:53 UTC, Tom Marshall
no flags Details

Description Stephen Tweedie 2006-09-21 10:04:13 UTC
Duplicate VG names strike again.  Except with Xen, it's worse.

The trouble is that we can have nested PVs inside a LV.  In fact, if you install
a Xen guest (or indeed _any_ virtualised image, be it Xen, qemu or whatever)
onto an LV, you're going to end up with a partition inside the LV disk image
which itself contains a PV; and if the user uses default VG naming, then that
nested PV is going to be initialised with the same VG name for every guest (and
the same name as the host's own VG if the user has not overridden this.)

Now, these nested VGs are active parts of a virtual image: there are boot
loaders on the disk image's /boot which refer to the VG by name. So renaming the
VG is not a viable procedure for maintenance purposes if an admin wants to
access the nested VG without breaking the image as a whole.

And it gets worse if an admin tries to snapshot the disk image (a reasonable
thing to do if you want to backup the xen image via LVM snapshots.)  In that
case, we end up with two separate LVs, each of which contains a nested PV with
the *SAME* UUID, but not necessarily the same contents --- these are not simply
multiple paths to the same data.

Now, it's probably reasonable to simply error out if we find such duplicate
UUIDs, as long as we do so cleanly --- a user trying to activate an origin and
its snapshot at the same time via kpartx/vgchange is asking for trouble.  But
preferable would be a way to activate a vggroup based on one of its PVs.

Even without the handling of duplicated UUIDs, a way to activate a VG and access
LVs from amongst multiple VGs with the same name is neede in order to give
administrative access to data hidden inside a domU.

Version-Release number of selected component (if applicable):
All

Comment 1 James Ralston 2007-07-24 22:00:59 UTC
I just got burned by this, and I'm not a happy camper.  :(

I'd be willing to settle for being able to temporarily rename the nested VG, but
you can't even do that:

$ lvs
  WARNING: Duplicate VG name os: Existing DxyAhi-pYiU-z8L7-UmzA-R0cF-ynic-HzdzES
(created here) takes precedence over 326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3
  LV     VG   Attr   LSize  Origin Snap%  Move Log Copy% 
  root   os   -wi-ao  4.00G                              
  usr    os   -wi-ao  8.00G                              
  ...

Ok, fine, I'll just temporarily rename the nested volume group to something else:

$ vgrename 326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3 xos
  WARNING: Duplicate VG name os: Existing DxyAhi-pYiU-z8L7-UmzA-R0cF-ynic-HzdzES
(created here) takes precedence over 326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3
  Volume group "os" still has active LVs

Okay, logical volumes (which lvs won't show me!) in the nested VG are active. 
I'll just deactivate the nested VG by referring to it by UUID:

$ vgchange -a n 326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3
  WARNING: Duplicate VG name os: Existing DxyAhi-pYiU-z8L7-UmzA-R0cF-ynic-HzdzES
(created here) takes precedence over 326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3
  Volume group "326MmU-VwzI-jMgG-stSe-4UFY-OcrX-1EcVO3" not found

Surprise, surprise: vgrename will accept a UUID for a VG name (even though this
behavior isn't documented anywhere I can find), but vgchange won't.

I'm sympathetic to the problem of handling duplicate VGs (because damn, is that
a thorny problem), but there's no excuse for inconsistencies in the applications
which will accept UUIDs as sources or targets.  Pick a scheme that makes sense
(e.g.: any name/path argument of the form "uuid:blah-blah-blah-blah" is looked
up by UUID), implement that scheme consistently, and document it.


Comment 2 Bug Zapper 2008-04-03 18:16:54 UTC
Based on the date this bug was created, it appears to have been reported
against rawhide during the development of a Fedora release that is no
longer maintained. In order to refocus our efforts as a project we are
flagging all of the open bugs for releases which are no longer
maintained. If this bug remains in NEEDINFO thirty (30) days from now,
we will automatically close it.

If you can reproduce this bug in a maintained Fedora version (7, 8, or
rawhide), please change this bug to the respective version and change
the status to ASSIGNED. (If you're unable to change the bug's version
or status, add a comment to the bug and someone will change it for you.)

Thanks for your help, and we apologize again that we haven't handled
these issues to this point.

The process we're following is outlined here:
http://fedoraproject.org/wiki/BugZappers/F9CleanUp

We will be following the process here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping to ensure this
doesn't happen again.

Comment 3 Mark McLoughlin 2008-04-04 06:42:00 UTC
Still very much relevant AFAIK

Comment 4 Bug Zapper 2008-05-14 02:22:12 UTC
Changing version to '9' as part of upcoming Fedora 9 GA.
More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 5 Dave Wysochanski 2008-06-04 18:01:52 UTC
Steven,

I think I understand what you are saying but let me try to summarize.

You would like the ability for an administrator to access LVs on an embedded
domU image file, so you can get at the data on there without having to boot the
domU (which may not even be possible in some cases).  I in fact ran into this
recently as well when an install of rawhide on a domU failed to install grub
properly.  I was able to work around this by using a procedure similar to this:
http://wiki.xensource.com/xenwiki/InstallGuestImage
and by renaming the volume group.  As I recall I hit the bug of the device file
not getting properly updated on the rename, and I will check the latest code for
the bug in comment #1.  Did I summarize the original problem correctly?

Will think about ways to improve the situation so maintenance of xen domU images
are simpler with LVM tools.  Not sure what the answer is - might go beyond LVM
tool enhancement.

Comment 6 James Ralston 2008-06-05 16:37:11 UTC
The bug in comment #1 has been fixed for at least RHEL 5.1 and 5.2; vgchange now
accepts a UUID to specify the target volume group.  So, to get at the volumes of
a virtual host, you can use kpartx/vgscan/vgchange/vgrename in sequence.

The primary way I've avoided this problem is by naming the main volume group
"os" for the Xen host, and naming the main volume group "vos" for Xen guests. 
That way, I only have to rename volume groups if I want to access the
filesystems of two guests simultaneously, which occurs very infrequently. 
(Wanting to access the filesystem of a single powered-down guest is much more
common.)

With that in mind, here's a simple suggestion: if anaconda detects that it is
occurring within a paravirtualized install, or if the hardware it detects in the
system looks suspiciously like the hardware Xen would present to an HVM guest,
use a different automatic LVM volume naming scheme than "VolGroup00" (e.g.,
"VVolGroup00".)  Or consider using /dev/urandom to generate a portion of the
suggested volume group names randomly.

Not coping well with duplicate volume group names strikes me as more of a
misfeature of LVM than an outright bug, so I have my doubts as to whether it
will be possible to have LVM really "play nice" with duplicate volume group
names.  (Mounting the filesystems of other hosts probably wasn't that common of
an operation until virtualization came along...)


Comment 7 Dave Wysochanski 2008-06-05 18:09:29 UTC
Thanks for the info and great suggestions!

CC-ing Peter Jones and will ping him separately to see what he thinks about
detecting virtualized setups and modifying the default vol group name for guests.

Comment 8 Stephen Tweedie 2008-06-06 16:24:44 UTC
> The bug in comment #1 has been fixed for at least RHEL 5.1 and 5.2; vgchange now
> accepts a UUID to specify the target volume group.  So, to get at the volumes of
> a virtual host, you can use kpartx/vgscan/vgchange/vgrename in sequence.

That works in dire emergency, but has the disadvantage of rendering the guest
unbootable in the process --- if you change the guest vgname, the initrd won't
be able to find the root filesystem.

> if anaconda detects that it is
> occurring within a paravirtualized install, or if the hardware it detects in the
> system looks suspiciously like the hardware Xen would present to an HVM guest,
> use a different automatic LVM volume naming scheme than "VolGroup00" (e.g.,
> "VVolGroup00".)  Or consider using /dev/urandom to generate a portion of the
> suggested volume group names randomly.

That would be ideal, but I think it came up before and was rejected by the
installer folks.  It's also too late for existing installs, of course; and we
still need to be able to support installs of old OS versions that don't do this.



Comment 9 Dave Wysochanski 2008-08-29 20:07:23 UTC
One potential option that is relatively easy and gets us a little further is this.  Modify vgrename to have a "--force" option on it and allow renames into duplicate VG names.

I don't entirely like this idea, but it's relatively simple to implement in the code from what I can tell.  I'm not sure of all potential side-effects yet (e.g. cluster?)  The change is not ideal, but allows us to do a sequence like the following.

1. Use vgrename to map the duplicate xen domU VG (e.g. VolGroup00) to a temporary name, using the 'uuid' of the VG
2. Activate the VG, mount LVs, do maintenance, etc
3. Unmount the LVs, deactivate the VG
4. Rename the VG back to the original name.

Currently we allow vgrename away from duplicate VG names, but not back.  The current situation is not good because you can get yourself into a situation where you can't get back to a bootable system very easily (step #4 above will fail).

I am looking at alternatives of using the UUID to disambiguate the LV / dm target names but right now that is looking more involved.

Comment 10 Dave Wysochanski 2008-09-03 17:46:00 UTC
From what I can tell, below is what we have today in the virtualization guide for dealing with guest images:
http://tinyurl.com/5pstgn

Comment 11 Stephen Tweedie 2008-09-08 13:41:43 UTC
That virtualisation-guide recipe is precisely what won't work with default volume group naming conventions.

As for renaming volume groups, yes, that may provide one solution.  But as noted it does risk leaving an image unbootable; and as a general principle of system diagnosis/recovery, users should not have to significantly reconfigure their disks just in order to be able to perform maintenance on them.

Comment 12 Dave Wysochanski 2008-09-08 18:23:18 UTC
Agreed - I've got a setup myself and have been playing with maintenance on my own guest images so I feel the pain!

I'm looking into better solutions such as the code detecting duplicate names and perhaps add something (part of the UUID?) to remove the ambiguity.  Making such an LV usable (for maintenance) would require modification of the device mapper namespace but not the on-disk metadata, so this needs some thought and maybe a special option.

Comment 13 Bug Zapper 2008-11-26 01:50:36 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 10 development cycle.
Changing version to '10'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 15 Ask Bjørn Hansen 2009-04-17 08:13:36 UTC
"One potential option that is relatively easy and gets us a little further is
this.  Modify vgrename to have a "--force" option on it and allow renames into
duplicate VG names."

Just wanted to second how useful that would be.  As others have mentioned, getting proper "duplicate VG name" handling seems, well, more of a long term solution.

Just being able to force rename a vg would make it possible to save otherwise unbootable images without having to copy the image over to a box with a different vg name...

Comment 16 Ask Bjørn Hansen 2009-04-17 08:44:18 UTC
For the "my xen image is fubar'ed and I need to mess with the data to make it boot" situation I just realized an obvious-ish work-around:  Run a new box with the rescue image.

Something like

virt-install -n box1b -r 1024 -f /xen/disks/brokendisk  -p -l http://mirrors/5/os/x86_64/ -d --vnc --vcpus=1 -x rescue 

...

Anyway, it just saved me a re-install.   (Of course the one box that isn't managed completely by puppet etc was the one to go kaboom).

 - ask

Comment 17 Bug Zapper 2009-06-09 09:10:00 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 11 development cycle.
Changing version to '11'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 18 Bug Zapper 2009-11-16 07:52:43 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 19 Guymon 2010-01-28 21:20:50 UTC
"One potential option that is relatively easy and gets us a little further is
this.  Modify vgrename to have a "--force" option on it and allow renames into
duplicate VG names."

I'll second that!

My story:
After doing a vgrename, I was able to mount it and go in and fix some files, but on doing vgrename to set it back to VolGroup00, I ran into this problem... Arrrggggg.

Here's my bad remedy all from Dom0, it prevented me from rebuilding from scratch. This worked for me, so do at your own risk.  I'm also breezing over stuff quick as if you got yourself in this mess, you should be pretty conferable as a sysadmin.

My mess up started with
I had vgrename VolGroup00 to --> VolGroup00_WET

To make it work again in Xen We have to change the grub.conf and the initrd.img (my case initrd-2.6.18-164.10.1.el5xen.img) along with the /etc/fstab as it will try to fsck VolGroup00 and fail on boot.

1st. We need to mount the boot partition with the initrd*.img and the grub.conf files.
I used: 
mount -o loop,offset=32256 /dev/VolGroup_Wet/LogVol_Wet /mnt/LogVol_WET (love offset!)

/mnt/LogVol_WET being my mnt space on Dom0.

In here we will find /grub/grub.conf. Vim in and change the VolGroup00 to VolGroup00_WET(what my NEW VolGroup got vgrename too) Next grab the initrd*.img. My current one was called initrd-2.6.18-164.10.1.el5xen.img.  I found out how to change this here [http://www.fedoraforum.org/forum/showthread.php?t=114819] Thanks!!!! heres a quick copy paste from there.
---------------------------------------------------------------------------
the problem is that the intrd was indeed in the gzip format and further in the cpio format, but the cpio format is not the default "bin" format used by cpio but a "newc" format, so the extraction part was perfect, but when you put it back you change the cpio format and hence the kernel hangs

(collection of files and directories) -> cpio archives -> gzip -> final initrd.img

that means you can give the following commands -
mkdir ~/tmp
cd ~/tmp
cp /boot/initrd.img ./initrd.gz
gunzip initrd.gz
mkdir tmp2
cd tmp2
cpio -id < ../initrd.img
now you should have a lot of files in ~/tmp/tmp2 directories, including a lot of subdirectories like sbin,lib

now do the required changes to the files
then pack the files back into the archive using the following command
cd ~/tmp/tmp2

find . | cpio --create --format='newc' > ~/tmp/newinitrd
cd ~/tmp
gzip newinitrd

now you would have a newinitrd.gz
rename this now -
mv newinitrd.gz as newinitrd.img
this is the new boot image now !! 
----------------------------------------------------------------------------
Move this new image back to your mounted partition.  I renamed the old one, just incase.

Whew thats done! unmount.
umount /mnt/LogVol_WET

2nd. Lets Mount root /  . Unfortunately offset won't work on this.
I did this like this from the Dom0.

/sbin/kpartx -a /dev/VolGroup_Wet/LogVol_Wet
/usr/sbin/pvscan
/usr/sbin/lvscan
/usr/sbin/vgchange -a y
## Mount it on /mnt/LogVol_WET
mount /dev/VolGroup00_WET/LogVol00 /mnt/LogVol_WET

now go in vim /mnt/LogVol_WET/etc/fstab and change VolGroup00 to VolGroup00_WET(My new volgrp)
DON'T edit your Dom0 fstab.. I did that but luckily noticed and fixed my mistake!!!!

phew... I think that's everything. Now unmount and clean up your kpartx stuff

umount /mnt/LogVol_WET
/usr/sbin/vgchange -a n
/sbin/kpartx -d /dev/VolGroup_Wet/LogVol_Wet

Now I can 
/usr/sbin/xm create WET 
and I'm golden!

I know this seems like a lot and I learned a lot.  I mostly put this here knowing I will loose my notes some time in the future. You have to love Google.  I would have been hosed!

Oh yea as I state before all this could have been avoided if vgrename had a "--force" option!!!!!!

Comment 20 Guymon 2010-01-28 21:31:36 UTC
Ooops forgot a step.

After you open the initrd img you need to vim in and edit the init file

in here at the bottom you will find references to the /dev/VolGroup00/LogVol00 change to /dev/VolGroup00_WET/LogVol00 (of as needed)

Sorry Kind of important.  I didn't know how to edit my comment above.

Comment 21 Richard W.M. Jones 2010-01-28 21:32:19 UTC
(In reply to comment #19)
> After doing a vgrename, I was able to mount it and go in and fix some files,
> but on doing vgrename to set it back to VolGroup00, I ran into this problem...
> Arrrggggg.

Sounds complicated though.  For editing stuff in your
Xen disk images, use http://libguestfs.org/

Comment 22 Bug Zapper 2010-07-30 10:28:50 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 14 development cycle.
Changing version to '14'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 23 Tom Marshall 2010-08-17 21:53:50 UTC
Created attachment 439221 [details]
Tool to edit PV fields, including VG name

This is a quick hack to work around the duplicate VG name issue.  It is not intended for general use, only as an emergency tool to fix issues as described in this bug report.

If the VG is contained in a single PV, it may be renamed with eg. "pvtool /dev/nbd0p1 vg_name=vg00_tmp".

Comment 24 Milan Broz 2010-08-17 22:17:31 UTC
That tool jst handles lvm1 format - which is deprected and all new installations use new text metadata format.

Anyway, you can rename VG using its UUID already- see "man vgrename"
vgrename Zvlifi-Ep3t-e0Ng-U42h-o0ye-KHu1-nl7Ns4 VolGroup00_tmp
(this should work for lvm1 format too.)

Comment 25 Peter Rajnoha 2011-11-10 16:07:55 UTC
LVM device filtering could be used here to make use of some problematic devices, using the "--config" option for LVM2 commands, like:

  --config 'devices { filter = [ "r|/dev/sda|...|" ] }'

..where you can give the list of devices which should be filtered out (devices used as PVs where a duplicate VG is). Then you can manage the problematic VG/LV. However, you can't rely on activation to work since you could already have the duplicate VG activated of course (we would need to do something like Dave suggested already in comment #12, however, I think this is a no-go for now).

As for supporting the --force option to allow renaming to a duplicate name as mentioned in comment #9 - we could probably add support for this... But it's really not very nice.

As for using the libguestfs to access the contents of the image (as mentioned in comment #21), this seems to be the most elegant solution I think (together with defining a filter directly in /etc/lvm/lvm.conf to exclude all LVM devices used as guest images so any LVM setup inside won't interfere with the host LVM setup). However, we need to take into account that libguestfs runs a new VM instance with it's own minimalistic kernel/rootfs and the image we need to access added.

I think we could go with the filters set + libguestfs for now. Would that be satisfactory?

(Just a note, that there's also an RFE filed for using UUID on input of other LVM2 commands: bug #449832)

Comment 26 Peter Rajnoha 2011-11-10 16:21:13 UTC
(In reply to comment #25)
> I think we could go with the filters set + libguestfs for now.

+ guestfish

Comment 27 Richard W.M. Jones 2011-11-10 16:31:01 UTC
In case it's not clear from comment 26, libguestfs supports
adjusting the LVM filters:

http://libguestfs.org/guestfs.3.html#guestfs_lvm_set_filter
http://libguestfs.org/guestfs.3.html#guestfs_lvm_clear_filter

We use it in virt-df already.

Comment 28 Jaroslav Kortus 2012-03-27 17:51:46 UTC
I found another way how to mount images which (may) have duplicate vg names:

1. setup loopback device with the name
2. setup loopback device for data you are going to write to it (must have)
3. use device mapper to map loopback device  to dm target, read-only (will allow us to use snapshots)
4. create writeable snapshot of the device from 3.
5. use kpartx to map partitions to new dm devices (kpartx -a <device from 4.>)
6. use vgcloneimport -n my_not_colliding_vg_name /dev/mapper/jkdev_snap2 (replace with output from 5)
7. pvscan, lvscan, vgchange ... all that fun you wish to do with the newly created VG and its LVs

The reason for using this is that vgcloneimport destroys the mapping and corrupts the image (no longer bootable without further changes) by changing UUIDs/names. This way the changes go right to the snapshot and keeps the original image intact.

If you wish to write to the original image while you are doing backup, it would be better to create proper snapshot with snapshot-origins as described in http://www.redhat.com/archives/dm-devel/2004-July/msg00071.html (use dm create ... --notable). For shutdown&examine (my case) this is probably the most simple way.

Clean up via dmsetup remove <device> and losetup -d.

# sample script:
imgfile="/var/lib/libvirt/images/node02.img"
snapfile="/var/lib/libvirt/images/snapshot.img"

devname="jkdev"
snap="jkdev_snap"

loop1="/dev/loop1"
loop2="/dev/loop2"

# loop device
# map loop1 -> image file
# map loop2 -> snapshot file
losetup -r $loop1 $imgfile
dd if=/dev/zero of=$snapfile bs=1M seek=100 count=0
losetup $loop2 $snapfile

# map dm -> loop1 
dmsetup -r create $devname --table "0 `blockdev --getsize $loop1` linear $loop1 0"

# snapshot mappings
dmsetup create $snap --table "0 `blockdev --getsize /dev/mapper/$devname` snapshot /dev/mapper/$devname $loop2 p 64 "

Comment 29 Fedora End Of Life 2013-04-03 19:54:49 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 19 development cycle.
Changing version to '19'.

(As we did not run this process for some time, it could affect also pre-Fedora 19 development
cycle bugs. We are very sorry. It will help us with cleanup during Fedora 19 End Of Life. Thank you.)

More information and reason for this action is here:
https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora19

Comment 30 Fedora End Of Life 2015-01-09 21:35:42 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 31 Fedora End Of Life 2015-02-18 11:56:26 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.