Bug 221613

Summary: restoring failed mirrors appears to be broken
Product: Red Hat Enterprise Linux 4 Reporter: Corey Marthaler <cmarthal>
Component: lvm2Assignee: Jonathan Earl Brassow <jbrassow>
Status: CLOSED WORKSFORME QA Contact: Corey Marthaler <cmarthal>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4CC: agk, dwysocha, jbrassow, mbroz, prockai, rkenna, slevine
Target Milestone: ---Keywords: Documentation
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2007-12-20 17:18:57 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Corey Marthaler 2007-01-05 17:08:30 UTC
Description of problem:

I started with a simple mirror.

[root@link-08 ~]# lvcreate  -m 1 -n mirror -L 5G vg /dev/sda1:0-1300
/dev/sdb1:0-1300 /dev/sdc1:0:100
  Logical volume "mirror" created

[root@link-08 ~]# lvs -a -o +devices
  LV                VG   Attr   LSize Origin Snap%  Move Log         Copy%  Devices
  mirror            vg   mwi-a- 5.00G                    mirror_mlog 100.00
mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] vg   iwi-ao 5.00G                                      
/dev/sda1(0)
  [mirror_mimage_1] vg   iwi-ao 5.00G                                      
/dev/sdb1(0)
  [mirror_mlog]     vg   lwi-ao 4.00M                                      
/dev/sdc1(0)

I then failed the primary leg (/dev/sda1) by turning off power to that device.

[root@link-08 ~]# lvs -a -o +devices
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda: read failed after 0 of 4096 at 1296954228736: Input/output error
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 1024 at 648472756224: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 512 at 648472756224: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 5368643584: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  Volume group "vg" not found


I then did a write to that mirror volume inorder to cause lvm to down convert
that volume to a usable linear.

[root@link-08 ~]# dd if=/dev/zero of=/dev/vg/mirror count=10
10+0 records in
10+0 records out
[root@link-08 ~]# lvscan
  /dev/sda: read failed after 0 of 4096 at 0: Input/output error
  /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sda2: read failed after 0 of 2048 at 0: Input/output error
  /dev/dm-3: read failed after 0 of 4096 at 0: Input/output error
  ACTIVE            '/dev/vg/mirror' [5.00 GB] inherit

I then turned that device back on with the hopes of being able to resotre that
mirror back to it's original state.

[root@link-08 ~]# cat /proc/partitions
major minor  #blocks  name

   3     0   78150744 hda
   3     1     104391 hda1
   3     2   78043770 hda2
   8     0 1266556928 sda
   8     1  633274267 sda1
   8     2  633274267 sda2
   8    16  142255575 sdb
   8    17  142247542 sdb1
   8    32  142255575 sdc
   8    33  142247542 sdc1
   8    48  142255575 sdd
   8    49  142247542 sdd1
   8    64  142255575 sde
   8    65  142247542 sde1
   8    80  142255575 sdf
   8    81  142247542 sdf1
   8    96  142255575 sdg
   8    97  142247542 sdg1
   8   112  142255575 sdh
   8   113  142247542 sdh1
 253     0   75956224 dm-0
 253     1    2031616 dm-1
 253     5    5242880 dm-5

But it doesn't show up as a pv yet

[root@link-08 ~]# pvscan
  Warning: Volume Group vg is not consistent
  PV /dev/sdb1   VG vg   lvm2 [135.66 GB / 130.66 GB free]
  PV /dev/sdc1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdh1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]

So I then attempted an lvmdiskscan, not sure if that was necessary but I figured
it wouldn't hurt.

lvm> lvmdiskscan
  /dev/ram0  [       16.00 MB]
  /dev/root  [       72.44 GB]
  /dev/ram   [       16.00 MB]
  /dev/sda1  [      603.94 GB] LVM physical volume
  /dev/dm-1  [        1.94 GB]
  /dev/ram2  [       16.00 MB]
  /dev/sda2  [      603.94 GB] LVM physical volume
  /dev/ram3  [       16.00 MB]
  /dev/ram4  [       16.00 MB]
  /dev/ram5  [       16.00 MB]
  /dev/dm-5  [        5.00 GB]
  /dev/ram6  [       16.00 MB]
  /dev/ram7  [       16.00 MB]
  /dev/ram8  [       16.00 MB]
  /dev/ram9  [       16.00 MB]
  /dev/ram10 [       16.00 MB]
  /dev/ram11 [       16.00 MB]
  /dev/ram12 [       16.00 MB]
  /dev/ram13 [       16.00 MB]
  /dev/ram14 [       16.00 MB]
  /dev/ram15 [       16.00 MB]
  /dev/sdb1  [      135.66 GB] LVM physical volume
  /dev/sdc1  [      135.66 GB] LVM physical volume
  /dev/sdd1  [      135.66 GB] LVM physical volume
  /dev/sde1  [      135.66 GB] LVM physical volume
  /dev/sdf1  [      135.66 GB] LVM physical volume
  /dev/sdg1  [      135.66 GB] LVM physical volume
  /dev/sdh1  [      135.66 GB] LVM physical volume
  2 disks
  17 partitions
  0 LVM physical volume whole disks
  9 LVM physical volumes
lvm> pvscan
  Warning: Volume Group vg is not consistent
  PV /dev/sdb1   VG vg   lvm2 [135.66 GB / 130.66 GB free]
  PV /dev/sdc1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdh1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  Total: 7 [949.59 GB] / in use: 7 [949.59 GB] / in no VG: 0 [0   ]

So it show up as an lvm pv though lvmdiskscan but still not though pvscan

Now I attempted to up convert this volume back to a mirror but this operation failed

lvm> lvconvert -m 1 /dev/vg/mirror
  Inconsistent metadata copies found - updating to use version 10
  Volume group vg metadata is inconsistent
  Volume group for uuid not found:
EV34uF3M1qqikowj1ozGfH3dFhohehehhZTonEPkRcGcynjzVrJo7w0ZC6kU4V0k
  Aborting. Failed to activate mirror log. Remove new LVs and retry.
  Failed to create mirror log.
[root@link-08 ~]# lvscan
  Volume group "vg" inconsistent
  Inconsistent metadata copies found - updating to use version 12
  ACTIVE            '/dev/vg/mirror' [5.00 GB] inherit
  inactive          '/dev/vg/mirror_mlog' [4.00 MB] inherit
[root@link-08 ~]# lvs -a -o +devices
  Volume group "vg" inconsistent
  Inconsistent metadata copies found - updating to use version 13
  LV          VG   Attr   LSize Origin Snap%  Move Log Copy%  Devices
  mirror      vg   -wi-a- 5.00G                               /dev/sdb1(0)
  mirror_mlog vg   -wi--- 4.00M                               /dev/sdh1(0)
[root@link-08 ~]# dmsetup ls
vg-mirror       (253, 5)
VolGroup00-LogVol01     (253, 1)
VolGroup00-LogVol00     (253, 0)
[root@link-08 ~]# dmsetup table
vg-mirror: 0 10485760 linear 8:17 384
VolGroup00-LogVol01: 0 4063232 linear 3:2 151912832
VolGroup00-LogVol00: 0 151912448 linear 3:2 384
[root@link-08 ~]# dmsetup ls --tree
vg-mirror (253:5)
 └─ (8:17)
VolGroup00-LogVol01 (253:1)
 └─ (3:2)
VolGroup00-LogVol00 (253:0)
 └─ (3:2)


So either this operation is very broken or I'm not doing it correctly and the
correct way to restore a mirror needs to be documented.


Version-Release number of selected component (if applicable):
2.6.9-42.17.ELsmp
lvm2-2.02.17-1

How reproducible:
everytime

Comment 1 Corey Marthaler 2007-01-05 20:48:53 UTC
This turns out to be a documentation issue, in that there is a way to make this
work, but I just wasn't sure how. Here is what needs to happen inorder to
restore the mirror...

When the failed disk comes back (or in this case is powered back up) and lvm
notices it, it complains of an inconsisant vg, that is because that disk was
appart of the old vg and knew that it had more devices in it earlier, where as
those other devices currently in the vg are fine because they has basically
written that failed device off. So now what needs to happen is for the failed
device to be treated as a "new" device and recreated as a new pv, and then
extended to the consistent vg, and then the mirror can be up converted.


[root@link-08 ~]# pvcreate /dev/sda1
  Warning: Volume group vg is not consistent
  Physical volume "/dev/sda1" successfully created
[root@link-08 ~]# pvcreate /dev/sda2
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  Couldn't find device with uuid 'TGIIFC-15fe-vyMd-0mIN-Dmi4-iDD7-IPda53'.
  Couldn't find all physical volumes for volume group vg.
  Warning: Volume group vg is not consistent
  Physical volume "/dev/sda2" successfully created
[root@link-08 ~]# vgextend vg /dev/sda1
  Volume group "vg" successfully extended
[root@link-08 ~]# vgextend vg /dev/sda2
  Volume group "vg" successfully extended
[root@link-08 ~]# pvscan
  PV /dev/sdb1   VG vg   lvm2 [135.66 GB / 130.66 GB free]
  PV /dev/sdc1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdd1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sde1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdf1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdg1   VG vg   lvm2 [135.66 GB / 135.66 GB free]
  PV /dev/sdh1   VG vg   lvm2 [135.66 GB / 135.65 GB free]
  PV /dev/sda1   VG vg   lvm2 [603.93 GB / 603.93 GB free]
  PV /dev/sda2   VG vg   lvm2 [603.93 GB / 603.93 GB free]
  Total: 9 [2.11 TB] / in use: 9 [2.11 TB] / in no VG: 0 [0   ]
[root@link-08 ~]# lvconvert -m 1 /dev/vg/mirror
  Logical volume mirror converted.
[root@link-08 ~]# lvs -a -o +devices
  LV                VG   Attr   LSize Origin Snap%  Move Log         Copy%  Devices
  mirror            vg   mwi-a- 5.00G                    mirror_mlog 100.00
mirror_mimage_0(0),mirror_mimage_1(0)
  [mirror_mimage_0] vg   iwi-ao 5.00G                                      
/dev/sdb1(0)
  [mirror_mimage_1] vg   iwi-ao 5.00G                                      
/dev/sda1(0)
  [mirror_mlog]     vg   lwi-ao 4.00M                                      
/dev/sdh1(0)


Comment 2 Alasdair Kergon 2007-01-05 20:54:54 UTC
As well as documenting this, for a future release let's try to handle this
better - there are definitely things we could do in the logic - but we've got
plenty of higher priority matters to deal with first.

Comment 4 Alasdair Kergon 2007-02-07 16:39:09 UTC
Patch added upstream for 2.02.22 which spots when a PV that had disappeared and
been removed from the VG (with vgreduce --removemissing) reappears and wipes the
VG metadata on it so it no longer thinks it is part of the VG.

Comment 5 Michael Hideo 2007-06-06 04:41:56 UTC
Adding 'cc ecs-dev-list for tracking

Comment 6 Jonathan Earl Brassow 2007-08-31 19:39:50 UTC
# killed sdb - the secondary leg
# I did the following commands after the down convert

[root@neo-04 ~]# lvs
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  36.62G
  LogVol01 VolGroup00 -wi-ao 512.00M
  lv       vg         -wi-a- 500.00M
[root@neo-04 ~]# ./on.sh sdb
Turning on sdb
[root@neo-04 ~]# lvs
  LV       VG         Attr   LSize   Origin Snap%  Move Log Copy%
  LogVol00 VolGroup00 -wi-ao  36.62G
  LogVol01 VolGroup00 -wi-ao 512.00M
  lv       vg         -wi-a- 500.00M
[root@neo-04 ~]# pvs
  PV         VG         Fmt  Attr PSize  PFree
  /dev/hda2  VolGroup00 lvm2 a-   37.16G 32.00M
  /dev/sda   vg         lvm2 a-   17.15G 16.66G
  /dev/sdc   vg         lvm2 a-   17.15G 17.15G
  /dev/sdd   vg         lvm2 a-   17.15G 17.15G
  /dev/sde   vg         lvm2 a-   17.15G 17.15G

We good now?


Comment 7 Michael Hideo 2007-10-23 02:49:06 UTC
Removing automation notification

Comment 8 Corey Marthaler 2007-12-20 17:18:57 UTC
Closing this bug...