Bug 908097 - Cannot vgimport repair exported broken LVM mirror
Cannot vgimport repair exported broken LVM mirror
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: lvm2 (Show other bugs)
5.8
All Linux
high Severity high
: rc
: ---
Assigned To: Jonathan Earl Brassow
Cluster QE
: Patch
Depends On:
Blocks: 836232 928849 908381
  Show dependency treegraph
 
Reported: 2013-02-05 16:28 EST by Dave Sullivan
Modified: 2013-09-30 20:27 EDT (History)
13 users (show)

See Also:
Fixed In Version: lvm2-2.02.88-11.el5
Doc Type: Bug Fix
Doc Text:
Previously, if a device had failed after a vgexport was issued, it would be impossible to import the volume group. Additionally, a failure to import also meant it was impossible to repair the volume group. It is now possible to use the '--force' option with vgimport to import volume groups even if there are devices missing.
Story Points: ---
Clone Of:
: 908381 (view as bug list)
Environment:
Last Closed: 2013-09-30 20:27:52 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Dave Sullivan 2013-02-05 16:28:18 EST
Description of problem:

If you have an exported mirrored LVM DG that is missing a mirror it is impossible to import the DG.  Tested in 5.8 and 6.3... See following reproducer:

[root@ll-chsystest03 logs]# dd if=/dev/zero of=loop0 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.583542 s, 1.8 GB/s

[root@ll-chsystest03 logs]# dd if=/dev/zero of=loop1 bs=1M count=1000
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 0.593067 s, 1.8 GB/s

[root@ll-chsystest03 logs]# losetup /dev/loop0 /logs/loop0

[root@ll-chsystest03 logs]# losetup /dev/loop1 /logs/loop1

[root@ll-chsystest03 logs]# pvcreate /dev/loop0

[root@ll-chsystest03 logs]# pvcreate /dev/loop1

[root@ll-chsystest03 logs]# vgcreate TRASHME /dev/loop0 /dev/loop1

[root@ll-chsystest03 logs]# lvcreate --contiguous n --alloc normal -m 1 -n trashme -L500m TRASHME /dev/loop0 /dev/loop1
  Logical volume "trashme" created

[root@ll-chsystest03 logs]# lvs
  LV      VG      Attr     LSize   Pool Origin Data%  Move Log          Copy%  Convert
  trashme TRASHME mwn-aom- 500.00m                         trashme_mlog 100.00
  logs    vg0     -wi-ao-- 520.69g
  root    vg0     -wi-ao--  14.00g
  tmp     vg0     -wi-ao--   2.00g
  var     vg0     -wi-ao--   6.00g


[root@ll-chsystest03 logs]# vgs
  VG      #PV #LV #SN Attr   VSize   VFree
  TRASHME   2   1   0 wz--n-   1.95g 988.00m
  vg0       1   4   0 wz--n- 542.69g      0


[root@ll-chsystest03 logs]# vgchange -an TRASHME
  0 logical volume(s) in volume group "TRASHME" now active

[root@ll-chsystest03 logs]# vgexport TRASHME
  Volume group "TRASHME" successfully exported

[root@ll-chsystest03 logs]# losetup -d /dev/loop1

[root@ll-chsystest03 logs]# vgs
  Couldn't find device with uuid utPEIg-JC7U-4xP0-BXsu-oRl9-IjHS-4oyEGQ.
  VG      #PV #LV #SN Attr   VSize   VFree
  TRASHME   2   1   0 wzxpn-   1.95g 988.00m
  vg0       1   4   0 wz--n- 542.69g      0


[root@ll-chsystest03 logs]# vgimport TRASHME
  Couldn't find device with uuid utPEIg-JC7U-4xP0-BXsu-oRl9-IjHS-4oyEGQ.
  Cannot change VG TRASHME while PVs are missing.
  Consider vgreduce --removemissing.

[root@ll-chsystest03 logs]# vgreduce  TRASHME --removemissing --force
  Couldn't find device with uuid utPEIg-JC7U-4xP0-BXsu-oRl9-IjHS-4oyEGQ.
  Unable to determine mirror sync status of TRASHME/trashme.
  Volume group "TRASHME" is exported
  Failed to lock trashme

The "Failed to lock" problem is where the issue comes in.  "vgreduce" cannot fix the problem because it can't get a lock on an exported diskgroup, and vgimport can't import the diskgroup since its "broken".  Catch-22.



Version-Release number of selected component (if applicable):

RHEL5.8 and RHEL6.3

How reproducible:


See above


Additional info:

Workaround #1

#this will give me the pv uuid
vgdisplay  --partial --verbose

cd /etc/lvm/backup

There should be a file there called TRASHME

get a pv same size as what was there before, your are simulating

losetup /dev/loop1 /logs/loop1

now restore the pv with same uuid and meta data from our /etc/lvm/backup/TRASHME file
pvcreate --restorefile TRASHME --uuid mBR3gH-V90u-yOOk-V4BL-Llg0-UFH1-MfKbib /dev/loop1 -ff

[root@localhost backup]# vgimport TRASHME
  Volume group "TRASHME" successfully imported

Workaround #2

You could hand edit the backup file of the LVM metadata (generally found in /etc/lvm/backup/<VG NAME>).  You would simply need to remove the 'EXPORTED' flag wherever it is found and then run vgcfgrestore
Comment 3 Jonathan Earl Brassow 2013-04-10 16:26:43 EDT
This bug is the same as bug 913644.

It was fixed with the following upstream commit:
https://bugzilla.redhat.com/show_bug.cgi?id=913644#c2

commit 3ab46449f4dbcc45fe878149838a8439f5ac8b34
Author: Jonathan Brassow <jbrassow@redhat.com>
Date:   Wed Feb 20 16:28:26 2013 -0600

    vgimport:  Allow '--force' to import VGs with missing PVs.
    
    When there are missing PVs in a volume group, most operations that alter
    the LVM metadata are disallowed.  It turns out that 'vgimport' is one of
    those disallowed operations.  This is bad because it creates a circular
    dependency.  'vgimport' will complain that the VG is inconsistent and that
    'vgreduce --removemissing' must be run.  However, 'vgreduce' cannot be run
    because it has not been imported.  Therefore, 'vgimport' must be one of
    the operations allowed to change the metadata when PVs are missing.  The
    '--force' option is the way to make 'vgimport' happen in spite of the
    missing PVs.
Comment 5 Nenad Peric 2013-05-16 06:03:44 EDT
Verified that import can succeeed with suppliting --force with 
lvm2-2.02.88-11.el5

(04:59:32) [root@r5-node02:~]$ vgimport trashed
  Couldn't find device with uuid 6QXgTN-nbEw-kTvO-2IR7-s0tS-RwlG-GvGV9f.
  Cannot change VG trashed while PVs are missing.
  Consider vgreduce --removemissing.
(04:59:42) [root@r5-node02:~]$ vgimport trashed --force
  '--force' supplied.  Volume groups with missing PVs will be imported.
  Couldn't find device with uuid 6QXgTN-nbEw-kTvO-2IR7-s0tS-RwlG-GvGV9f.
  Volume group "trashed" successfully imported
(04:59:55) [root@r5-node02:~]$ vgs
  Couldn't find device with uuid 6QXgTN-nbEw-kTvO-2IR7-s0tS-RwlG-GvGV9f.
  VG         #PV #LV #SN Attr   VSize  VFree  
  VolGroup00   1   2   0 wz--n-  8.88G      0 
  test         7   1   0 wz--n- 69.97G  69.48G
  trashed      2   1   0 wz-pn-  1.95G 988.00M
Comment 7 Nenad Peric 2013-05-21 09:16:35 EDT
Marking verified with lvm2-2.02.88-11.el5

(08:12:18) [root@r5-node02:~]$ losetup -d /dev/loop1 
(08:12:39) [root@r5-node02:~]$ vgs
  Couldn't find device with uuid oEuL7I-FJfA-kg0P-gbRm-VOUq-FAyE-Mc17Kn.
  VG         #PV #LV #SN Attr   VSize  VFree  
  VolGroup00   1   2   0 wz--n-  8.88G      0 
  test         4   3   0 wz--n- 39.98G  39.39G
  trashme      2   1   0 wzxpn-  1.95G 988.00M
(08:12:43) [root@r5-node02:~]$ vgimport trashme
  Couldn't find device with uuid oEuL7I-FJfA-kg0P-gbRm-VOUq-FAyE-Mc17Kn.
  Cannot change VG trashme while PVs are missing.
  Consider vgreduce --removemissing.
(08:12:53) [root@r5-node02:~]$ vgimport trashme --force
  '--force' supplied.  Volume groups with missing PVs will be imported.
  Couldn't find device with uuid oEuL7I-FJfA-kg0P-gbRm-VOUq-FAyE-Mc17Kn.
  Volume group "trashme" successfully imported
(08:12:56) [root@r5-node02:~]$ vgs
  Couldn't find device with uuid oEuL7I-FJfA-kg0P-gbRm-VOUq-FAyE-Mc17Kn.
  VG         #PV #LV #SN Attr   VSize  VFree  
  VolGroup00   1   2   0 wz--n-  8.88G      0 
  test         4   3   0 wz--n- 39.98G  39.39G
  trashme      2   1   0 wz-pn-  1.95G 988.00M
Comment 8 errata-xmlrpc 2013-09-30 20:27:52 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1352.html

Note You need to log in before you can comment on or make changes to this bug.