Bug 1268867 - Unable to remove VG with duplicate VG name under lvmetad without calling direct pvscan --cache even after the other VG is removed
Unable to remove VG with duplicate VG name under lvmetad without calling dire...
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
rawhide
All Linux
unspecified Severity medium
: ---
: ---
Assigned To: David Teigland
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-05 09:28 EDT by Peter Rajnoha
Modified: 2015-11-23 09:46 EST (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-11-23 09:46:37 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Peter Rajnoha 2015-10-05 09:28:35 EDT
When using lvmetad and we happen to end up in a state where we have 2 VGs with the same name (different VG UUIDs), we can remove one of the VGs, but not the other one then:

# vgcreate vg /dev/sda
  Volume group "vg" successfully created
# echo offline > /sys/block/sda/device/state
# pvscan --cach
# vgcreate vg /dev/sdb
  Physical volume "/dev/sdb" successfully created
  Volume group "vg" successfully created
# echo running > /sys/block/sda/device/state
# pvscan --cache
  WARNING: Duplicate VG name vg: Existing N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N (created here) takes precedence over VTJ5sZ-XM8j-IFqQ-quit-DfT9-Ofjt-eQQCI1
# pvs -o+vg_uuid
  PV         VG     Fmt  Attr PSize   PFree   VG UUID                               
  /dev/sda   vg     lvm2 a--  124.00m 124.00m N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N
  /dev/sdb   vg     lvm2 a--  124.00m 124.00m VTJ5sZ-XM8j-IFqQ-quit-DfT9-Ofjt-eQQCI1
# vgs -o+vg_uuid
  VG     #PV #LV #SN Attr   VSize   VFree   VG UUID                               
  vg       1   0   0 wz--n- 124.00m 124.00m VTJ5sZ-XM8j-IFqQ-quit-DfT9-Ofjt-eQQCI1
  vg       1   0   0 wz--n- 124.00m 124.00m N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N
# vgs -o+vg_uuid
  VG     #PV #LV #SN Attr   VSize   VFree   VG UUID                               
  fedora   1   2   0 wz--n-  19.49g      0  elQNcv-AmqS-nZb4-mro1-RBq1-9ojN-sIozB9
  vg       1   0   0 wz--n- 124.00m 124.00m N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N
# vgremove -ff vg
  Volume group "vg" not found
  Cannot process volume group vg
[0] f22/~ # vgs -o+vg_uuid
  VG     #PV #LV #SN Attr   VSize   VFree   VG UUID                               
  vg       1   0   0 wz--n- 124.00m 124.00m N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N

However, we can do that if we call pvscan --cache:
[0] f22/~ # pvscan --cache
[0] f22/~ # vgremove -ff vg
  Volume group "vg" successfully removed


This works without lvmetad:
(...creating the reproducing scenario as above with lvmetad...)

# vgs
  WARNING: Duplicate VG name vg: Existing yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI (created here) takes precedence over LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd
  WARNING: Duplicate VG name vg: Existing yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI (created here) takes precedence over LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd
  WARNING: Duplicate VG name vg: Existing LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd (created here) takes precedence over yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI
  WARNING: Duplicate VG name vg: Existing yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI (created here) takes precedence over LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd
  WARNING: Duplicate VG name vg: Existing LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd (created here) takes precedence over yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI
  VG     #PV #LV #SN Attr   VSize   VFree  
  vg       1   0   0 wz--n- 124.00m 124.00m
  vg       1   0   0 wz--n- 124.00m 124.00m

# vgremove -ff vg
  WARNING: Duplicate VG name vg: Existing yl3oJ7-FdKj-5bwq-FN7H-iFTf-QNCB-0oOkXI (created here) takes precedence over LO3sRe-7IlD-fG56-p2aS-cm45-P3q1-AKMbkd
  Volume group "vg" successfully removed

# vgremove -ff vg
  Volume group "vg" successfully removed
Comment 1 Peter Rajnoha 2015-10-05 10:29:26 EDT
(In reply to Peter Rajnoha from comment #0)
> # vgs -o+vg_uuid
>   VG     #PV #LV #SN Attr   VSize   VFree   VG UUID                         
> 
>   vg       1   0   0 wz--n- 124.00m 124.00m
> VTJ5sZ-XM8j-IFqQ-quit-DfT9-Ofjt-eQQCI1
>   vg       1   0   0 wz--n- 124.00m 124.00m
> N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N


  One more "vgremove vg" here! (I missed to paste it here from my cmd line, sorry.)


> # vgs -o+vg_uuid
>   VG     #PV #LV #SN Attr   VSize   VFree   VG UUID                         
> 
>   fedora   1   2   0 wz--n-  19.49g      0 
> elQNcv-AmqS-nZb4-mro1-RBq1-9ojN-sIozB9
>   vg       1   0   0 wz--n- 124.00m 124.00m
> N4Ai94-6ufw-EZ9O-G3QO-4Iau-FYQf-2Gsy8N
> # vgremove -ff vg
>   Volume group "vg" not found
>   Cannot process volume group vg
Comment 2 David Teigland 2015-10-05 12:43:21 EDT
In the past I think we've said "use vgimportclone" or "use filters", and declined to adjust other lvm commands for duplicate vg names.  If that is still our answer, then I think lvm commands that modify a VG, like vgremove, should refuse to operate on a VG with a duplicate name, whether lvmetad is used or not, e.g.

# vgremove vg
WARNING: ...
Cannot modify a VG using a duplicate name.
Use vgimportclone or filters to eliminate duplicate VG names.


But, shouldn't we allow lvm commands to modify vgs with duplicate names if a uuid is given? e.g.

# vgremove vg
WARNING: ...
Cannot modify a VG using a duplicate name.
Use vgimportclone or filters to eliminate duplicate VG names, or
use --select to specify the uuid of the intended VG.

# vgremove -S uuid=XYZ
(success)
Comment 3 Peter Rajnoha 2015-10-06 02:53:17 EDT
(In reply to David Teigland from comment #2)
> In the past I think we've said "use vgimportclone" or "use filters", and
> declined to adjust other lvm commands for duplicate vg names.  If that is
> still our answer, then I think lvm commands that modify a VG, like vgremove,
> should refuse to operate on a VG with a duplicate name, whether lvmetad is
> used or not, e.g.
> 
> # vgremove vg
> WARNING: ...
> Cannot modify a VG using a duplicate name.
> Use vgimportclone or filters to eliminate duplicate VG names.
> 
> 

The situation might be a bit different here where we have the VG with same name, but different UUIDs (as opposed to situation where even the UUID is the same where we have 1:1 clone). But it depends. Anyway, it would be fine for the behaviour to be consistent here with and without lvmetad.

I haven't checked the code around yet, but it seems this is just a bug since lvmetad still sees the VG correctly, even if there are same names (vgs still displays it). I'm able to remove the first VG with same name, but not the other one even after I removed the first one (so there's no duplication anymore).

> But, shouldn't we allow lvm commands to modify vgs with duplicate names if a
> uuid is given? e.g.
> 
> # vgremove vg
> WARNING: ...
> Cannot modify a VG using a duplicate name.
> Use vgimportclone or filters to eliminate duplicate VG names, or
> use --select to specify the uuid of the intended VG.
> 
> # vgremove -S uuid=XYZ
> (success)

This works with -S|--select, of course. Given that we have the -S now, we can decide whether to ban commands from doing things based on names when there are duplicates seen, which would makes sense I think...
Comment 4 Peter Rajnoha 2015-10-06 04:36:50 EDT
OK, so the reason is that lvmetad keeps all the metadata for both VGs with the same name. The bug is in a way lvmetad keeps lookup tables for vgname --> vgid mappings.

For example:

# vgs -o vg_name,vg_uuid
  VG     VG UUID                               
  vg     Q8vzCe-iLlT-cz8H-eG6Q-FKRT-l83M-AsQ2c6
  vg     nhu8eS-Lc5b-FN4g-rA6R-5AK0-SSL2-pDbudv


And this is an excerpt from lvmetad dump:


# VG METADATA
Q8vzCe-iLlT-cz8H-eG6Q-FKRT-l83M-AsQ2c6 {
...
}

nhu8eS-Lc5b-FN4g-rA6R-5AK0-SSL2-pDbudv {
...
}

# VGID to VGNAME mapping
vgid_to_vgname {
     Q8vzCe-iLlT-cz8H-eG6Q-FKRT-l83M-AsQ2c6 = "vg"
     nhu8eS-Lc5b-FN4g-rA6R-5AK0-SSL2-pDbudv = "vg"
}

# VGNAME to VGID mapping
vgname_to_vgid {
     vg = "nhu8eS-Lc5b-FN4g-rA6R-5AK0-SSL2-pDbudv"
}


So we have missing vg = "Q8vzCe-iLlT-cz8H-eG6Q-FKRT-l83M-AsQ2c6" mapping in the "vgname to vgid" mapping table.

This is not such a huge issue with removing devices, but it is quite an issue in a scenario where someone attaches disk to the system where by chance the VG name is the same as the existing one already present in the system - lvmetad will get updated based on event (the pvscan --cache <device> call via udev) and then when such disk is detached, lvmetad loses info about the original VG.

There's a workaround for this though - to run pvscan --cache directly on command line for lvmetad to fixup its vgname to vgid mappings after the VG with same name is removed, but I can imagine this to be quite puzzling for users since all reporting commands display the VG exists while when trying to process it, LVM says it does not exist.
Comment 5 Peter Rajnoha 2015-10-06 04:38:50 EDT
(In reply to Peter Rajnoha from comment #4)
> This is not such a huge issue with removing devices

s/devices/VGs
Comment 6 Peter Rajnoha 2015-10-06 04:43:49 EDT
Also, vgrename! If I rename the duplicate VG (by using its UUID), I just lose the mapping from vgame to vgid for the original VG.
Comment 7 David Teigland 2015-10-06 10:36:26 EDT
For the past couple of weeks I've been rewriting the lvmetad functions update_metadata() and pv_found(). That began while trying to fix vgchange -u which lvmetad cannot handle either.

The current behavior of duplicate vg names (with different uuids) appears to be more accidental than intentional.  In my new version I'll handle this case explicitly.  The way hash tables are used makes it rather diffuclt to handle duplicate VG names, just like it's difficult and ugly to handle duplicate PVs.

The core data structures (the hash tables) are too inflexible to handle all the special cases (i.e. duplicates).  I think we may need to store and organize the core information differently, using an optimization on the side to speed up lookups (hash tables or other).
Comment 8 Peter Rajnoha 2015-10-07 04:29:49 EDT
(In reply to David Teigland from comment #7)
> The core data structures (the hash tables) are too inflexible to handle all
> the special cases (i.e. duplicates).  I think we may need to store and
> organize the core information differently, using an optimization on the side
> to speed up lookups (hash tables or other).

I wonder if we could just use a list as hash table entry so whenever duplicates appear, all the values with same key would be put on the list (instead of single value as it is today which makes the duplicate to be lost).
Comment 9 David Teigland 2015-11-16 18:02:28 EST
lvmetad now handles multiple VGs with the same name independently.  Select can be used to choose which VG to operate on.  A command given just the VG name will fail, explaining that multiple VGs exist with that name.  Are there commands where we want to just pick one instead?
Comment 10 Ondrej Kozina 2015-11-23 04:05:04 EST
David, if so feel free to close the bug since it was originally intended to track the lvemtad issue which was probably taken care of already. Also we can open new bug to track required changes in lvm commands if there are any...

Note You need to log in before you can comment on or make changes to this bug.