Bug 1730520
Summary: | if we have 2 vg's with the same name, after renaming one, it uses the other's PV's to get metadata! | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | nikhil kshirsagar <nkshirsa> | ||||||||||
Component: | lvm2 | Assignee: | David Teigland <teigland> | ||||||||||
lvm2 sub component: | Changing Logical Volumes | QA Contact: | cluster-qe <cluster-qe> | ||||||||||
Status: | CLOSED NOTABUG | Docs Contact: | |||||||||||
Severity: | high | ||||||||||||
Priority: | unspecified | CC: | agk, heinzm, jbrassow, jmagrini, msnitzer, prajnoha, teigland, zkabelac | ||||||||||
Version: | 7.6 | ||||||||||||
Target Milestone: | rc | ||||||||||||
Target Release: | --- | ||||||||||||
Hardware: | Unspecified | ||||||||||||
OS: | Unspecified | ||||||||||||
Whiteboard: | |||||||||||||
Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
Doc Text: | Story Points: | --- | |||||||||||
Clone Of: | Environment: | ||||||||||||
Last Closed: | 2020-01-28 15:03:36 UTC | Type: | Bug | ||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||
Documentation: | --- | CRM: | |||||||||||
Verified Versions: | Category: | --- | |||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
Embargoed: | |||||||||||||
Attachments: |
|
Created attachment 1591241 [details]
metadata from 1 of the 5 nodes
Created attachment 1591254 [details]
metadata from node jdb02
Created attachment 1591265 [details]
verbose outputs of some lvm commands
> At this point use_lvmetad was set to 1 in lvm.conf , they do not use clvmd > > for this 5 node RAC cluster but told us that > > " > > These are manœuvre LUNs (shared LUNs), it’s supposed to be created / > declared on ALL servers but mounted on only 1 server at once. > This must not be mounted on multiple servers in the same time, for sure. > This will be used to copy DB export from 1 server (A) to another (B) using > below process : > FS /manoeuvre1 and /manoeuvre2 are mounted on server A > DB export is copied to FS /manoeuvre1 and /manoeuvre2 on server A > FS /manoeuvre1 and /manoeuvre2 are unmounted on server A > FS /manoeuvre1 and /manoeuvre2 are mounted on server B > DB export is copied from FS /manoeuvre1 and /manoeuvre2 on server B > FS /manoeuvre1 and /manoeuvre2 are unmounted on server B This is highly incorrect and improper use of lvm. The customer should: - Disable lvmetad immediately and permanently. lvmetad is incompatible with shared storage; the problems they have seen already are only the beginning. - Enable system ID on every host that can see the shared storage, and assign system IDs to each VG on the shared storage. See lvmsystemid(7) for more information about using system ID. - To move the VG between servers, run vgexport on the source server, then run vgimport on the destination server. - If they want different servers to use LVs in a single VG simultaneously, the only supported way to do this is with lvmlockd or clvmd. > # vgrename uuid manoeuvrevg1_bkp
>
> "
>
> On hindsight it may have been better to do a vgremove here instead of a
> rename?
Once you have disabled lvmetad and are using system IDs, then duplicate VG names are only a problem if both VGs are visible on a single host. If that's the case, then the only thing you can do to resolve it is vgrename <uuid> <newname>, vgremove will not be allowed (it will not know which VG you mean to remove.)
> So what they requested was, to know how to remove the duplicated manoeuvrevg1
>
> we want to remove that extra VG(which is duplicated ) in this server.
>
> VG Attr Ext #PV #LV #SN VSize VFree VG UUID
> manoeuvrevg1 wz-pn- 4.00m 15 1 0 <2.93t 0
> g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch
> manoeuvrevg1 wz-pn- 4.00m 15 1 0 <2.93t 0
> XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj
> manoeuvrevg2 wz-pn- 4.00m 15 1 0 <2.93t 0
> FtAV6z-TlT1-fpBU-bfsQ-3t0f-Eq8V-v2z64W
What do you mean by "want to remove"? Do you want it to not be visible, or do you want to wipe the disks, or do you want it to be visible with a different name?
Whichever case it is, to start repairing things you need to first detach the storage from all but one host, and disable lvmetad everywhere. Then you need to figure out which devices (/dev/sdX) belong to each of the different VGs. Then you need to set up the filter in lvm.conf to exclude all of the devices from the VG you are not going to work on. Then try to repair and rename the VG on the remaining devices (or wipe them if you don't want that VG to exist any more.) Then change the filter to exclude the devices you've just repaired or wiped, and include the devices you excluded before, and repeat the process of repairing or wiping the VG on those devices.
|
Created attachment 1591239 [details] vgscan verbose Description of problem: The customer opened a support case asking to remove a VG with a duplicate name. This probably happened because he attached a disk which had metadata (containing a vg with the same name as a vg currently present on this server) So what they requested was, to know how to remove the duplicated manoeuvrevg1 we want to remove that extra VG(which is duplicated ) in this server. VG Attr Ext #PV #LV #SN VSize VFree VG UUID manoeuvrevg1 wz-pn- 4.00m 15 1 0 <2.93t 0 g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch manoeuvrevg1 wz-pn- 4.00m 15 1 0 <2.93t 0 XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj manoeuvrevg2 wz-pn- 4.00m 15 1 0 <2.93t 0 FtAV6z-TlT1-fpBU-bfsQ-3t0f-Eq8V-v2z64W At this point use_lvmetad was set to 1 in lvm.conf , they do not use clvmd for this 5 node RAC cluster but told us that " These are manœuvre LUNs (shared LUNs), it’s supposed to be created / declared on ALL servers but mounted on only 1 server at once. This must not be mounted on multiple servers in the same time, for sure. This will be used to copy DB export from 1 server (A) to another (B) using below process : FS /manoeuvre1 and /manoeuvre2 are mounted on server A DB export is copied to FS /manoeuvre1 and /manoeuvre2 on server A FS /manoeuvre1 and /manoeuvre2 are unmounted on server A FS /manoeuvre1 and /manoeuvre2 are mounted on server B DB export is copied from FS /manoeuvre1 and /manoeuvre2 on server B FS /manoeuvre1 and /manoeuvre2 are unmounted on server B " Support then asked them to do this. " You have 2 VG with the same name: VG Attr Ext #PV #LV #SN VSize VFree VG UUID manoeuvrevg1 wz-pn- 4.00m 15 0 0 <2.93t 0 g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch manoeuvrevg1 wz--n- 4.00m 15 1 0 <2.93t 0 XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj Action Plan: o Either remove the luns which are not used by active VG. o Or, Change the Active VG name to something else like manoeuvrevg1_bkp with the help of uuid. # vgrename uuid manoeuvrevg1_bkp " On hindsight it may have been better to do a vgremove here instead of a rename? After this, what we ended up with was, VG #PV #LV #SN Attr VSize VFree VG UUID applivg 3 3 0 wz--n- <599.99g <402.99g xVR2VB-tfeh-nZdH-BQhw-h6Xr-GrW1-So5rzG manoeuvrevg1 15 0 0 wz-pn- <2.93t <2.93t g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch manoeuvrevg1_bkp 15 1 0 wz--n- <2.93t 0 XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj manoeuvrevg2 15 1 0 wz--n- <2.93t 0 Ghrsc1-a3Rp-AvOx-flNz-WxUS-LbTt-BXfu5Z oraclevg 1 6 0 wz--n- <200.00g <53.00g 2vZ1mF-d6BA-aE7S-R929-B00x-mdh1-wb05Tx rootvg 1 8 0 wz--n- <205.39g 187.91g D92t9d-1VOd-fBYy-ErdZ-cvp1-1J0G-VrugJY [root@gerdcwora1pdb01 ~]$ [root@gerdcwora1pdb01 ~]$ [root@gerdcwora1pdb01 ~]$ vgremove g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch manoeuvrevg1 Volume group "g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch" not found Cannot process volume group g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch Couldn't find device with uuid HwdVE1-qX25-afEh-826D-jsTz-BhYP-czmphj. Couldn't find device with uuid YgZMMp-E3bj-seOn-sZ51-yj8G-uI2p-7n8oCf. Couldn't find device with uuid IdCvLh-qhcS-3DVM-WAfd-pKEs-H13P-0wGy3Q. Couldn't find device with uuid XdLxSW-Lqys-gOTm-etDG-gn0Q-ZikX-YYiZee. Couldn't find device with uuid 3YzWwB-OrUC-VObv-OfVm-oqQ5-kp5h-q3Jrhb. Couldn't find device with uuid qX3OWD-SbON-uhNf-IJkw-PFOU-02Z0-WU9LNc. Couldn't find device with uuid 4l7vq9-ofaQ-omTS-au8v-EHdG-dqS1-YGSCpy. Couldn't find device with uuid P7kxSI-Srgj-ATMM-DkDL-a4Xh-S2Al-YEKlZu. Couldn't find device with uuid qc68ka-bTwE-ia11-SIAL-5pfL-hi10-Fe0CfV. Couldn't find device with uuid ITcdBU-EGyz-cx0W-OBCN-6Pbe-4dg2-84s3qo. Couldn't find device with uuid PhgD0G-ay2R-7e0h-4ZBj-z45Q-ptd8-ywUO0i. Couldn't find device with uuid WDfaFC-ldPa-pbub-QdCY-m8kb-yIK9-j8Byzu. Couldn't find device with uuid Td7uJO-nUld-Xm4r-oTzO-15zG-McRk-Ti5D0X. Couldn't find device with uuid q78InL-XaEU-yLC1-ef6r-cU37-ZyVp-y13Ibj. Couldn't find device with uuid 8sVcjf-fl0V-kHZX-MAOl-AIt1-ggl8-z0Ywns. Volume group "manoeuvrevg1" not found, is inconsistent or has PVs missing. Consider vgreduce --removemissing if metadata is inconsistent. [root@gerdcwora1pdb01 ~]$ [root@gerdcwora1pdb01 ~]$ vgremove manoeuvrevg1 Couldn't find device with uuid HwdVE1-qX25-afEh-826D-jsTz-BhYP-czmphj. Couldn't find device with uuid YgZMMp-E3bj-seOn-sZ51-yj8G-uI2p-7n8oCf. Couldn't find device with uuid IdCvLh-qhcS-3DVM-WAfd-pKEs-H13P-0wGy3Q. Couldn't find device with uuid XdLxSW-Lqys-gOTm-etDG-gn0Q-ZikX-YYiZee. Couldn't find device with uuid 3YzWwB-OrUC-VObv-OfVm-oqQ5-kp5h-q3Jrhb. Couldn't find device with uuid qX3OWD-SbON-uhNf-IJkw-PFOU-02Z0-WU9LNc. Couldn't find device with uuid 4l7vq9-ofaQ-omTS-au8v-EHdG-dqS1-YGSCpy. Couldn't find device with uuid P7kxSI-Srgj-ATMM-DkDL-a4Xh-S2Al-YEKlZu. Couldn't find device with uuid qc68ka-bTwE-ia11-SIAL-5pfL-hi10-Fe0CfV. Couldn't find device with uuid ITcdBU-EGyz-cx0W-OBCN-6Pbe-4dg2-84s3qo. Couldn't find device with uuid PhgD0G-ay2R-7e0h-4ZBj-z45Q-ptd8-ywUO0i. Couldn't find device with uuid WDfaFC-ldPa-pbub-QdCY-m8kb-yIK9-j8Byzu. Couldn't find device with uuid Td7uJO-nUld-Xm4r-oTzO-15zG-McRk-Ti5D0X. Couldn't find device with uuid q78InL-XaEU-yLC1-ef6r-cU37-ZyVp-y13Ibj. Couldn't find device with uuid 8sVcjf-fl0V-kHZX-MAOl-AIt1-ggl8-z0Ywns. Volume group "manoeuvrevg1" not found, is inconsistent or has PVs missing. Consider vgreduce --removemissing if metadata is inconsistent. [root@gerdcwora1pdb01 ~]$ We asked them to disable lvmetad, and try pvscan,vgscan, but they continued seeing this problem on 4 nodes out of 5. 1 "good" node in this 5 node cluster doesn't have this issue, it sees, => On gerdcwora1pdb03 VG Attr Ext #PV #LV #SN VSize VFree VG UUID VProfile #VMda VMdaFree VMdaSize #VMdaUse VG Tags applivg wz--n- 4.00m 3 0 0 <599.99g <599.99g 7l3Dp6-w60R-ZP1t-tGzc-0jSX-kuls-5VXATh 3 0 1020.00k 3 manoeuvrevg1 wz--n- 4.00m 15 1 0 <2.93t 0 XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj 15 0 1020.00k 15 manoeuvrevg2 wz--n- 4.00m 15 1 0 <2.93t 0 Ghrsc1-a3Rp-AvOx-flNz-WxUS-LbTt-BXfu5Z 15 0 1020.00k 15 oraclevg wz--n- 4.00m 1 6 0 <200.00g <87.00g IB3hGl-VPpB-RZhL-dH2O-lqKN-ah6l-SApuH0 1 0 1020.00k 1 rootvg wz--n- 4.00m 1 8 0 <205.39g <188.89g xiVCaF-AhkL-0W1q-LNih-UvvL-GQ2s-6i4iq8 1 0 1020.00k 1 Here the problematic VG is not seen. On the bad nodes, with ALL MISSING pv's, somehow the VG seemed partially activated, and vgscan seems to indicate its picking up the metadata from the PV meant for the VG we renamed to manoeuvrevg1_bpk. [root@nkshirsa sosreport-gerdcwora1jdb01-2019-07-16-mgqibpm]# cat sos_commands/lvm2/vgs_-v_-o_vg_mda_count_vg_mda_free_vg_mda_size_vg_mda_used_count_vg_tags_--config_global_locking_type_0 Reloading config files WARNING: Locking disabled. Be careful! This could corrupt your metadata. Couldn't find device with uuid HwdVE1-qX25-afEh-826D-jsTz-BhYP-czmphj. Couldn't find device with uuid YgZMMp-E3bj-seOn-sZ51-yj8G-uI2p-7n8oCf. Couldn't find device with uuid IdCvLh-qhcS-3DVM-WAfd-pKEs-H13P-0wGy3Q. Couldn't find device with uuid XdLxSW-Lqys-gOTm-etDG-gn0Q-ZikX-YYiZee. Couldn't find device with uuid 3YzWwB-OrUC-VObv-OfVm-oqQ5-kp5h-q3Jrhb. Couldn't find device with uuid qX3OWD-SbON-uhNf-IJkw-PFOU-02Z0-WU9LNc. Couldn't find device with uuid 4l7vq9-ofaQ-omTS-au8v-EHdG-dqS1-YGSCpy. Couldn't find device with uuid P7kxSI-Srgj-ATMM-DkDL-a4Xh-S2Al-YEKlZu. Couldn't find device with uuid qc68ka-bTwE-ia11-SIAL-5pfL-hi10-Fe0CfV. Couldn't find device with uuid ITcdBU-EGyz-cx0W-OBCN-6Pbe-4dg2-84s3qo. Couldn't find device with uuid PhgD0G-ay2R-7e0h-4ZBj-z45Q-ptd8-ywUO0i. Couldn't find device with uuid WDfaFC-ldPa-pbub-QdCY-m8kb-yIK9-j8Byzu. Couldn't find device with uuid Td7uJO-nUld-Xm4r-oTzO-15zG-McRk-Ti5D0X. Couldn't find device with uuid q78InL-XaEU-yLC1-ef6r-cU37-ZyVp-y13Ibj. Couldn't find device with uuid 8sVcjf-fl0V-kHZX-MAOl-AIt1-ggl8-z0Ywns. Couldn't find device with uuid HwdVE1-qX25-afEh-826D-jsTz-BhYP-czmphj. Couldn't find device with uuid YgZMMp-E3bj-seOn-sZ51-yj8G-uI2p-7n8oCf. Couldn't find device with uuid IdCvLh-qhcS-3DVM-WAfd-pKEs-H13P-0wGy3Q. Couldn't find device with uuid XdLxSW-Lqys-gOTm-etDG-gn0Q-ZikX-YYiZee. Couldn't find device with uuid 3YzWwB-OrUC-VObv-OfVm-oqQ5-kp5h-q3Jrhb. Couldn't find device with uuid qX3OWD-SbON-uhNf-IJkw-PFOU-02Z0-WU9LNc. Couldn't find device with uuid 4l7vq9-ofaQ-omTS-au8v-EHdG-dqS1-YGSCpy. Couldn't find device with uuid P7kxSI-Srgj-ATMM-DkDL-a4Xh-S2Al-YEKlZu. Couldn't find device with uuid qc68ka-bTwE-ia11-SIAL-5pfL-hi10-Fe0CfV. Couldn't find device with uuid ITcdBU-EGyz-cx0W-OBCN-6Pbe-4dg2-84s3qo. Couldn't find device with uuid PhgD0G-ay2R-7e0h-4ZBj-z45Q-ptd8-ywUO0i. Couldn't find device with uuid WDfaFC-ldPa-pbub-QdCY-m8kb-yIK9-j8Byzu. Couldn't find device with uuid Td7uJO-nUld-Xm4r-oTzO-15zG-McRk-Ti5D0X. Couldn't find device with uuid q78InL-XaEU-yLC1-ef6r-cU37-ZyVp-y13Ibj. Couldn't find device with uuid 8sVcjf-fl0V-kHZX-MAOl-AIt1-ggl8-z0Ywns. There are 15 physical volumes missing. VG Attr Ext #PV #LV #SN VSize VFree VG UUID VProfile #VMda VMdaFree VMdaSize #VMdaUse VG Tags applivg wz--n- 4.00m 3 3 0 <599.99g <446.99g prTh5H-qz53-22S9-WINL-zJ7U-Qc5J-ti5FRD 3 507.50k 1020.00k 3 manoeuvrevg1 wz-pn- 4.00m 15 0 0 <2.93t <2.93t g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch 1 506.00k 1020.00k 1 manoeuvrevg1_bkp wz--n- 4.00m 15 1 0 <2.93t 0 XOrFC0-rdEC-VjqY-Rdcj-Kemn-I88r-V9hgJj 14 504.00k 1020.00k 14 manoeuvrevg2 wz--n- 4.00m 15 1 0 <2.93t 0 Ghrsc1-a3Rp-AvOx-flNz-WxUS-LbTt-BXfu5Z 15 504.00k 1020.00k 15 oraclevg wz--n- 4.00m 1 7 0 <300.00g <154.00g QTYo03-fGma-O6yI-REy3-sk2C-f89M-gisLQJ 1 506.00k 1020.00k 1 rootvg wz--n- 4.00m 1 8 0 <205.39g <188.84g dMoyeX-Jd62-spdH-owqa-O1SU-BDUT-5zyX6j 1 506.00k 1020.00k 1 Reloading config files [root@nkshirsa sosreport-gerdcwora1jdb01-2019-07-16-mgqibpm]# What seems to have happened is, after the rename, the manoeuvrevg1 VG has picked up metadata from a PV belonging to the manoeuvrevg1_bkp VG. (/dev/mapper/manoeuvrevg1d1 ) Supporting data: in sosreport-gerdcwora1jdb01-2019-07-16-mgqibpm vgscan shows, Processing VG manoeuvrevg1 g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch <== Dropping cache for manoeuvrevg1. Reading VG manoeuvrevg1 g7ovuA-7dLz-xBH9-UMvX-rKkm-fXVv-rpC5Ch Rescanning devices for manoeuvrevg1 lvmcache has no info for vgname "manoeuvrevg1" with VGID g7ovuA7dLzxBH9UMvXrKkmfXVvrpC5Ch. Scanning 1 devices for VG info Scanning submitted 1 reads Processing data from device /dev/mapper/manoeuvrevg1d1 253:446 fd 4 block 0x559d34176b50 <== /dev/mapper/manoeuvrevg1d1: lvm2 label detected at sector 1 lvmcache /dev/mapper/manoeuvrevg1d1: now in VG #orphans_lvm2 (#orphans_lvm2) with 0 mda(s). /dev/mapper/manoeuvrevg1d1: PV header extension version 2 found Reading mda header sector from /dev/mapper/manoeuvrevg1d1 at 4096 Reading metadata summary from /dev/mapper/manoeuvrevg1d1 at 28672 size 3205 (+0) Found metadata summary on /dev/mapper/manoeuvrevg1d1 at 28672 size 3205 for VG manoeuvrevg1 lvmcache has no info for vgname "manoeuvrevg1" with VGID g7ovuA7dLzxBH9UMvXrKkmfXVvrpC5Ch. lvmcache has no info for vgname "manoeuvrevg1". lvmcache /dev/mapper/manoeuvrevg1d1: now in VG manoeuvrevg1 with 1 mda(s). lvmcache /dev/mapper/manoeuvrevg1d1: VG manoeuvrevg1: set VGID to g7ovuA7dLzxBH9UMvXrKkmfXVvrpC5Ch. lvmcache /dev/mapper/manoeuvrevg1d1: VG manoeuvrevg1: set seqno to 3 lvmcache /dev/mapper/manoeuvrevg1d1: VG manoeuvrevg1: set mda_checksum to 316583be mda_size to 3205 lvmcache /dev/mapper/manoeuvrevg1d1: VG manoeuvrevg1: set creation host to gerdcwora1jdb02. Scanned devices: read errors 0 process errors 0 failed 0 Reading VG manoeuvrevg1 from /dev/mapper/manoeuvrevg1d1 <----- this PV is of manoeuvrevg1_bkp Reading mda header sector from /dev/mapper/manoeuvrevg1d1 at 4096 Reading metadata from /dev/mapper/manoeuvrevg1d1 at 28672 size 3205 (+0) Allocated VG manoeuvrevg1 at 0x559d3843aa90. No device with uuid HwdVE1qX25afEh826DjsTzBhYPczmphj. Couldn't find device with uuid HwdVE1-qX25-afEh-826D-jsTz-BhYP-czmphj. No device with uuid YgZMMpE3bjseOnsZ51yj8GuI2p7n8oCf. Couldn't find device with uuid YgZMMp-E3bj-seOn-sZ51-yj8G-uI2p-7n8oCf. No device with uuid IdCvLhqhcS3DVMWAfdpKEsH13P0wGy3Q. Couldn't find device with uuid IdCvLh-qhcS-3DVM-WAfd-pKEs-H13P-0wGy3Q. No device with uuid XdLxSWLqysgOTmetDGgn0QZikXYYiZee. Couldn't find device with uuid XdLxSW-Lqys-gOTm-etDG-gn0Q-ZikX-YYiZee. note also: Found metadata on /dev/mapper/manoeuvrevg1d1 at 28672 size 3205 for VG manoeuvrevg1 Cached VG manoeuvrevg1 had incorrect PV list <---- Freeing VG manoeuvrevg1 at 0x559d3843aa90. Reading mda header sector from /dev/mapper/manoeuvrevg1d1 at 4096 Reading metadata from /dev/mapper/manoeuvrevg1d1 at 28672 size 3205 (+0) Allocated VG manoeuvrevg1 at 0x559d3843aa90. No device with uuid HwdVE1qX25afEh826DjsTzBhYPczmphj. The PV is this multipath device. manoeuvrevg1d1 (360000970000297900076533030303244) dm-446 EMC ,SYMMETRIX size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='service-time 0' prio=1 status=active |- 2:0:0:523 sdakk 132:960 active ready running |- 2:0:1:523 sdamy 8:1248 active ready running |- 16:0:0:523 sdaib 128:1008 active ready running `- 16:0:1:523 sdafi 68:896 active ready running As we see, its a PV for manoeuvrevg1_bkp. [root@nkshirsa sosreport-gerdcwora1jdb01-2019-07-16-mgqibpm]# cat sos_commands/lvm2/pvs_-a_-v_-o_pv_mda_free_pv_mda_size_pv_mda_count_pv_mda_used_count_pe_start_--config_global_locking_type_0 | grep manoeuvrevg1d1 /dev/mapper/manoeuvrevg1d1 manoeuvrevg1_bkp lvm2 a-- <200.00g 0 200.00g qUB1xp-Oi49-A000-yt6d-QJc8-rsB0-a4hH2R 506.00k 1020.00k 1 1 1.00m We are wondering how this has happened, could it be a bug in the rename? Would it be dangerous now to try vgremove --force of manoeuvrevg1, would it end up removing manoeuvrevg1_bkp since it seems to be pointing to the PV meant for manoeuvrevg1_bpk ! I will add more details in the additional details. Version-Release number of selected component (if applicable): lvm2-2.02.180-8.el7.x86_64 Mon Mar 4 14:10:27 2019 lvm2-libs-2.02.180-8.el7.x86_64 Mon Mar 4 14:10:26 2019 Additional info: Attaching vgscan -vvv to the bz. I will also attach lvm metadata in /etc/lvm