Attachment 915664 Details for Bug 905063 – Comment

Comment

comment_5456382.txt (text/plain), 80.62 KB, created by loberman on 2013-01-28 13:53:30 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: loberman

Created: 2013-01-28 13:53:30 UTC

Size: 80.62 KB

patch

obsolete

>Multiple issues with lvm2 and 2-disk mirrored configurations
>
>lvm2-libs-2.02.95-10.el6.x86_64
>lvm2-2.02.95-10.el6.x86_64
>
>How reproducible:
>
>Install RHEL 6.3, with multiple LVM volumes including swap, root, opt and var. Mirror to second device, disable or pull the mirror and multiple problems exist in recovery.
>
>Note even with only two volumes, manual intervention is required to remove "unkonwn device" from LVM group configuration.
>
>If var is included in the configuration, system hangs blocked in FS journal code. 
>In order to get var to be mirrored, we need to move /etc/lvm to a non LV mountpoint.
>Without var, mirror can be broken.
>In every case manual intervention is required to remove "unknown" device. If this is not done and system is rebooted, a panic or block in FSCK will occur as volumes are not accessible.
>
>In summary, the procedure (workaround) with manual intervention is not acceptable for an Enterprise customer. many weeks of efforts was put in to test multiple configurations.
>
>
>Steps to Reproduce:
>
>1. Install O/S with following configuration
>root, opt, var, home and swap in logical volumes
>
>
>2. Complete the install and then lvconvert the volume to a mirrored configuration with a second identical sized disk
>
>3. Let the mirrors sync to 100% and disable the mirror
>  
>Actual results:
>
>lvconvert -m1 vg_alain/lv_root --alloc anywhere
>lvconvert -m1 vg_alain/lv_opt --alloc anywhere
>lvconvert -m1 vg_alain/lv_var --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_oam --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_home --alloc anywhere
>
>
>Example tested
>
>[root@dhcp-189-249 lvm-tests]# lvs -a -o +devices
>  LV                 VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert Devices                                
>  lv_opt             vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00         lv_opt_mimage_0(0),lv_opt_mimage_1(0)  
>  [lv_opt_mimage_0]  vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(0)                           
>  [lv_opt_mimage_1]  vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(5000)                        
>  [lv_opt_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13001)                       
>  lv_root            vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00         lv_root_mimage_0(0),lv_root_mimage_1(0)
>  [lv_root_mimage_0] vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(5000)                        
>  [lv_root_mimage_1] vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(0)                           
>  [lv_root_mlog]     vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13000)                       
>  lv_swap            vg_alain     -wi-ao--  1.95g                                                     /dev/sda2(10000)                       
>  lv_var             vg_alain     mwi-aom-  9.77g                         lv_var_mlog  100.00         lv_var_mimage_0(0),lv_var_mimage_1(0)  
>  [lv_var_mimage_0]  vg_alain     iwi-aom-  9.77g                                                     /dev/sda2(10500)                       
>  [lv_var_mimage_1]  vg_alain     iwi-aom-  9.77g                                                     /dev/sdb2(10000)                       
>  [lv_var_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13002)                       
>  lv_home            vg_alain_app mwi-aom- 19.53g                         lv_home_mlog 100.00         lv_home_mimage_0(0),lv_home_mimage_1(0)
>  [lv_home_mimage_0] vg_alain_app iwi-aom- 19.53g                                                     /dev/sda3(0)                           
>  [lv_home_mimage_1] vg_alain_app iwi-aom- 19.53g                                                     /dev/sdb3(2500)                        
>  [lv_home_mlog]     vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7501)                        
>  lv_oam             vg_alain_app mwi-aom-  9.77g                         lv_oam_mlog  100.00         lv_oam_mimage_0(0),lv_oam_mimage_1(0)  
>  [lv_oam_mimage_0]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sda3(5000)                        
>  [lv_oam_mimage_1]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sdb3(0)                           
>  [lv_oam_mlog]      vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7500)           
>             
>
>Expected results:]
>device should be failed out of the volume group cleanly and system should not hang, even with var in an LVM mirror configuration.
>
>
>Additional info: 
>
>Lots of tests performed
>----------------------
>
>lvm -problem test 
>Start with --type mirror and 2 LV'S test and get a baseline
>
>
>[root@dhcp-189-249 mnt]# lvs -a -o +devices
>  LV                 VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert Devices                                
>  lv_opt             vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00         lv_opt_mimage_0(0),lv_opt_mimage_1(0)  
>  [lv_opt_mimage_0]  vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(0)                           
>  [lv_opt_mimage_1]  vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(5000)                        
>  [lv_opt_mlog]      vg_alain     lwi-aom-  4.00m                                                     /dev/sda2(13001)                       
>  lv_root            vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00         lv_root_mimage_0(0),lv_root_mimage_1(0)
>  [lv_root_mimage_0] vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(5000)                        
>  [lv_root_mimage_1] vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(0)                           
>  [lv_root_mlog]     vg_alain     lwi-aom-  4.00m                                                     /dev/sda2(13000)                       
>  lv_swap            vg_alain     -wi-ao--  1.95g                                                     /dev/sda2(10000)                       
>  lv_var             vg_alain     -wi-ao--  9.77g                                                     /dev/sda2(10500)                       
>  lv_home            vg_alain_app -wi-ao-- 19.53g                                                     /dev/sda3(0)                           
>  lv_oam             vg_alain_app -wi-ao--  9.77g                                                     /dev/sda3(5000)  
>
>Put lvm.conf mods here
>
>Captured to NFS mount
>Worked fine, rebooted but had to intervene to get it up.
>vgchange -ay --partial
>
>Then:
>  --- Physical volumes ---
>  PV Name               /dev/sda2     
>  PV UUID               707g3M-jZh4-jCQR-GSwO-NOlA-Fnj5-z2Vz25
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 1998
>   
>  PV Name               unknown device     
>  PV UUID               QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 9999
>   
>[root@dhcp-189-249 ~]# vgreduce --removemissing --force vg_alain
>  Couldn't find device with uuid QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH.
>  Wrote out consistent volume group vg_alain
>
>root@dhcp-189-249 ~]# vgreduce --removemissing vg_alain_app
>  Couldn't find device with uuid 7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF.
>  Wrote out consistent volume group vg_alain_app
>
>Put device back and get ready for next test
>
>------------------------------------------------------------------------------------------------
>
>
>add another LV to second disk group lv_alain_app and repeat the test
>and gather the same data
>
>[root@dhcp-189-249 lvm-tests]# lvs -a -o +devices  > lvs.mirror.3lv.pull
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid D8zndM-TNPZ-seD4-yNPi-lqJd-nL7N-W1L73t.
>
>^C  CTRL-c detected: giving up waiting for lock
>  /boot/var/lock/lvm/V_vg_alain: flock failed: Interrupted system call
>
>Remove lock file:
>
>[root@dhcp-189-249 lvm-tests]# ls -l /boot/var/lock/lvm/V_vg_alain
>-rwx------. 1 root root 0 Jan 18 10:32 /boot/var/lock/lvm/V_vg_alain
>[root@dhcp-189-249 lvm-tests]# rm /boot/var/lock/lvm/V_vg_alain
>rm: remove regular empty file `/boot/var/lock/lvm/V_vg_alain'? y
>[root@dhcp-189-249 lvm-tests]# lvs -a -o +devices  > lvs.mirror.3lv.pull
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid D8zndM-TNPZ-seD4-yNPi-lqJd-nL7N-W1L73t.
>  Couldn't find device with uuid 4U3RGk-lZ9k-utXB-iSvP-KoTz-2jey-VV7oVt.
>
>lvs stops hanging, byt vg commands hang such as vgdisplay -v
>
>Remove lock file again and vgdisplay passes..
>Unable to remove missing devices as it stops again on the lock file 
>
>Rebooting came up in good state after adding a disk back before rebooting.
>
>So this not a workable solution
>
>
>-----------------------------------------------------------------------------------
>
>NB --alloc anywhere is the kicker as well as /boot for lock
>
>Trying again
>
>lvconvert -m1 vg_alain/lv_root --alloc anywhere
>lvconvert -m1 vg_alain/lv_opt --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_oam --alloc anywhere
>
>
>pulled disk, 3 lvs command no issues
>cp /etc/hosts /root
>cp /etc/hosts /opt
>cp /etc/hosts /oam
>
>All work but we get disk errors
>
>Jan 18 12:31:03 localhost lvm[1457]: Mirror status: 1 of 2 images failed.
>Jan 18 12:31:03 localhost lvm[1457]: Repair of mirrored device vg_alain-lv_opt finished successfully.
>Jan 18 12:31:05 localhost lvm[1457]: No longer monitoring mirror device vg_alain-lv_opt for events.
>Jan 18 12:31:08 localhost kernel: end_request: I/O error, dev sdb, sector 132558848
>Jan 18 12:31:08 localhost lvm[1457]: Secondary mirror device 253:12 has failed (D).
>Jan 18 12:31:08 localhost lvm[1457]: Device failure in vg_alain_app-lv_oam.
>Jan 18 12:31:08 localhost lvm[1457]: /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>
>
>[root@dhcp-189-249 lvm-tests]# vgdisplay -v
>    Finding all volume groups
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    Finding volume group "vg_alain_app"
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  Couldn't find device with uuid CA3Mx3-3ZRU-Ww5d-CP6l-TMjA-D6Ik-wFkjNr.
>    There are 1 physical volumes missing.
>  --- Volume group ---
>  VG Name               vg_alain_app
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  22
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                2
>  Open LV               2
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               78.12 GiB
>  PE Size               4.00 MiB
>  Total PE              19998
>  Alloc PE / Size       7500 / 29.30 GiB
>  Free  PE / Size       12498 / 48.82 GiB
>  VG UUID               Ijnhwf-VdGk-kMJU-Plxl-orcd-ois1-JTWnCz
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_home
>  LV Name                lv_home
>  VG Name                vg_alain_app
>  LV UUID                rUde9I-AAvO-1bLF-mpqM-52hX-nrSX-T7Vsxa
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:32 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:5
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_oam
>  LV Name                lv_oam
>  VG Name                vg_alain_app
>  LV UUID                BWRJBP-1XSo-Mh5z-LBFl-NJ5X-c2p2-BpeFdp
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:33 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:9
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda3     
>  PV UUID               gdCaVp-wBWa-NgaC-J4v6-8i6I-Q3wE-UVzD5J
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 2499
>   
>  PV Name               unknown device     
>  PV UUID               CA3Mx3-3ZRU-Ww5d-CP6l-TMjA-D6Ik-wFkjNr
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 9999
>   
>    Finding volume group "vg_alain"
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  Couldn't find device with uuid crePPi-WeVw-TL3v-9xHX-iKpW-7siU-3ucMuE.
>    There are 1 physical volumes missing.
>  --- Volume group ---
>  VG Name               vg_alain
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  53
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                4
>  Open LV               4
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               117.18 GiB
>  PE Size               4.00 MiB
>  Total PE              29998
>  Alloc PE / Size       13000 / 50.78 GiB
>  Free  PE / Size       16998 / 66.40 GiB
>  VG UUID               TbtVxL-p0zk-yhX0-v1po-0CuP-LtxK-7U7y3f
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_opt
>  LV Name                lv_opt
>  VG Name                vg_alain
>  LV UUID                Vb73IK-H2Nh-7nZU-zH2x-tXcN-r7dv-Fb7Ss3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:26 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:13
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_root
>  LV Name                lv_root
>  VG Name                vg_alain
>  LV UUID                6ZPvoM-dHj3-P6Qd-WesB-VaSS-mOg2-xShaO3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:29 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:3
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_swap
>  LV Name                lv_swap
>  VG Name                vg_alain
>  LV UUID                yYebAk-sbn3-nLC7-VRcm-44Rc-3S3s-vuYbSZ
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:30 -0500
>  LV Status              available
>  # open                 1
>  LV Size                1.95 GiB
>  Current LE             500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:4
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_var
>  LV Name                lv_var
>  VG Name                vg_alain
>  LV UUID                dcSJaM-Gkjy-scbL-fL93-jdgc-cNtO-wrl9q4
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:31 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:14
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda2     
>  PV UUID               707g3M-jZh4-jCQR-GSwO-NOlA-Fnj5-z2Vz25
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 1999
>   
>  PV Name               unknown device     
>  PV UUID               crePPi-WeVw-TL3v-9xHX-iKpW-7siU-3ucMuE
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 14999
>
>So --alloc anywhere makes a difference
>
>Try and remove bad disks
>
>Seesm to work
>
>[root@dhcp-189-249 mnt]# vgreduce --removemissing --force vg_alain_app
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid CA3Mx3-3ZRU-Ww5d-CP6l-TMjA-D6Ik-wFkjNr.
>  Wrote out consistent volume group vg_alain_app
>
>[root@dhcp-189-249 mnt]# vgreduce --removemissing --force vg_alain    
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid crePPi-WeVw-TL3v-9xHX-iKpW-7siU-3ucMuE.
>  Wrote out consistent volume group vg_alain
>
>[root@dhcp-189-249 mnt]# vgdisplay -v
>    Finding all volume groups
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    Finding volume group "vg_alain_app"
>  --- Volume group ---
>  VG Name               vg_alain_app
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  23
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                2
>  Open LV               2
>  Max PV                0
>  Cur PV                1
>  Act PV                1
>  VG Size               39.06 GiB
>  PE Size               4.00 MiB
>  Total PE              9999
>  Alloc PE / Size       7500 / 29.30 GiB
>  Free  PE / Size       2499 / 9.76 GiB
>  VG UUID               Ijnhwf-VdGk-kMJU-Plxl-orcd-ois1-JTWnCz
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_home
>  LV Name                lv_home
>  VG Name                vg_alain_app
>  LV UUID                rUde9I-AAvO-1bLF-mpqM-52hX-nrSX-T7Vsxa
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:32 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:5
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_oam
>  LV Name                lv_oam
>  VG Name                vg_alain_app
>  LV UUID                BWRJBP-1XSo-Mh5z-LBFl-NJ5X-c2p2-BpeFdp
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:33 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:9
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda3     
>  PV UUID               gdCaVp-wBWa-NgaC-J4v6-8i6I-Q3wE-UVzD5J
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 2499
>   
>    Finding volume group "vg_alain"
>  --- Volume group ---
>  VG Name               vg_alain
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  54
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                4
>  Open LV               4
>  Max PV                0
>  Cur PV                1
>  Act PV                1
>  VG Size               58.59 GiB
>  PE Size               4.00 MiB
>  Total PE              14999
>  Alloc PE / Size       13000 / 50.78 GiB
>  Free  PE / Size       1999 / 7.81 GiB
>  VG UUID               TbtVxL-p0zk-yhX0-v1po-0CuP-LtxK-7U7y3f
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_opt
>  LV Name                lv_opt
>  VG Name                vg_alain
>  LV UUID                Vb73IK-H2Nh-7nZU-zH2x-tXcN-r7dv-Fb7Ss3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:26 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:13
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_root
>  LV Name                lv_root
>  VG Name                vg_alain
>  LV UUID                6ZPvoM-dHj3-P6Qd-WesB-VaSS-mOg2-xShaO3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:29 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:3
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_swap
>  LV Name                lv_swap
>  VG Name                vg_alain
>  LV UUID                yYebAk-sbn3-nLC7-VRcm-44Rc-3S3s-vuYbSZ
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:30 -0500
>  LV Status              available
>  # open                 1
>  LV Size                1.95 GiB
>  Current LE             500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:4
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_var
>  LV Name                lv_var
>  VG Name                vg_alain
>  LV UUID                dcSJaM-Gkjy-scbL-fL93-jdgc-cNtO-wrl9q4
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:31 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:14
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda2     
>  PV UUID               707g3M-jZh4-jCQR-GSwO-NOlA-Fnj5-z2Vz25
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 1999
>
>
>Reboot and see what happens
>
>Comes up clean
>Add a replacement disk and mirror again but add 4 mirrors
>
>lvconvert -m1 vg_alain/lv_root --alloc anywhere
>lvconvert -m1 vg_alain/lv_opt --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_oam --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_home --alloc anywhere
>
>
>  LV                 VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert Devices                                
>  lv_opt             vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00         lv_opt_mimage_0(0),lv_opt_mimage_1(0)  
>  [lv_opt_mimage_0]  vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(0)                           
>  [lv_opt_mimage_1]  vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(5000)                        
>  [lv_opt_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13001)                       
>  lv_root            vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00         lv_root_mimage_0(0),lv_root_mimage_1(0)
>  [lv_root_mimage_0] vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(5000)                        
>  [lv_root_mimage_1] vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(0)                           
>  [lv_root_mlog]     vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13000)                       
>  lv_swap            vg_alain     -wi-ao--  1.95g                                                     /dev/sda2(10000)                       
>  lv_var             vg_alain     -wi-ao--  9.77g                                                     /dev/sda2(10500)                       
>  lv_home            vg_alain_app mwi-aom- 19.53g                         lv_home_mlog 100.00         lv_home_mimage_0(0),lv_home_mimage_1(0)
>  [lv_home_mimage_0] vg_alain_app iwi-aom- 19.53g                                                     /dev/sda3(0)                           
>  [lv_home_mimage_1] vg_alain_app iwi-aom- 19.53g                                                     /dev/sdb3(2500)                        
>  [lv_home_mlog]     vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7501)                        
>  lv_oam             vg_alain_app mwi-aom-  9.77g                         lv_oam_mlog  100.00         lv_oam_mimage_0(0),lv_oam_mimage_1(0)  
>  [lv_oam_mimage_0]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sda3(5000)                        
>  [lv_oam_mimage_1]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sdb3(0)                           
>  [lv_oam_mlog]      vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7500)                        
>
>
>Pull disk
>
>Fails out properly
>
>Jan 18 12:54:24 localhost kernel: end_request: I/O error, dev sdb, sector 123906048
>Jan 18 12:54:24 localhost lvm[2377]: Mirror status: 1 of 2 images failed.
>Jan 18 12:54:24 localhost lvm[2377]: Repair of mirrored device vg_alain_app-lv_home finished successfully.
>Jan 18 12:54:26 localhost lvm[2377]: No longer monitoring mirror device vg_alain_app-lv_home for events.
>Jan 18 12:54:29 localhost kernel: __ratelimit: 496 callbacks suppressed
>Jan 18 12:54:29 localhost kernel: sd 2:0:1:0: [sdb] Unhandled error code
>Jan 18 12:54:29 localhost kernel: sd 2:0:1:0: [sdb] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
>Jan 18 12:54:29 localhost kernel: sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 07 66 d0 00 00 00 08 00
>Jan 18 12:54:29 localhost kernel: __ratelimit: 496 callbacks suppressed
>Jan 18 12:54:29 localhost lvm[2377]: Secondary mirror device 253:14 has failed (D).
>Jan 18 12:54:29 localhost lvm[2377]: Device failure in vg_alain_app-lv_oam.
>Jan 18 12:54:29 localhost lvm[2377]: /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>
>[root@dhcp-189-249 lvm-tests]# vgreduce --removemissing --force vg_alain
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid qwuyvC-B41g-o1Sv-aRPJ-PD6d-QRxw-j1vJiM.
>  Wrote out consistent volume group vg_alain
>
>[root@dhcp-189-249 lvm-tests]# vgreduce --removemissing --force vg_alain_app
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid hGHo32-LK2Q-Anys-oOCO-fxNI-hgBa-6f2gbX.
>  Wrote out consistent volume group vg_alain_app
>
>Remember /var is not mirrored
>
>Reboot to make sure 
>
>Now fix all 
>
>[root@dhcp-189-249 ~]# dd if=/dev/sda of=/dev/sdb bs=512b count=1
>1+0 records in
>1+0 records out
>262144 bytes (262 kB) copied, 0.00751753 s, 34.9 MB/s
>[root@dhcp-189-249 ~]# partprobe
>
>root@dhcp-189-249 ~]# pvcreate /dev/sdb2
>  Writing physical volume data to disk "/dev/sdb2"
>  Physical volume "/dev/sdb2" successfully created
>[root@dhcp-189-249 ~]# pvcreate /dev/sdb3
>  Writing physical volume data to disk "/dev/sdb3"
>  Physical volume "/dev/sdb3" successfully created
>[root@dhcp-189-249 ~]# vgextend vg_alain /dev/sdb2
>  Volume group "vg_alain" successfully extended
>[root@dhcp-189-249 ~]# vgextend vg_alain_app /dev/sdb3
>  Volume group "vg_alain_app" successfully extended
>
>Add /var
>
>lvconvert -m1 vg_alain/lv_root --alloc anywhere
>lvconvert -m1 vg_alain/lv_opt --alloc anywhere
>lvconvert -m1 vg_alain/lv_var --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_oam --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_home --alloc anywhere
>
>[root@dhcp-189-249 lvm-tests]# ls -l /boot/var
>total 2
>drwx------. 3 root root 1024 Jan 18 09:33 lock
>[root@dhcp-189-249 lvm-tests]# ls -l /boot/var/lock
>total 2
>drwx------. 2 root root 1024 Jan 18 13:04 lvm
>[root@dhcp-189-249 lvm-tests]# ls -l /boot/var/lock/lvm
>total 0
>
>[root@dhcp-189-9 lvm-tests]# cat create-mirrors.sh 
>lvconvert -m1 vg_alain/lv_root --alloc anywhere
>lvconvert -m1 vg_alain/lv_opt --alloc anywhere
>lvconvert -m1 vg_alain/lv_var --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_oam --alloc anywhere
>lvconvert -m1 vg_alain_app/lv_home --alloc anywhere
>
>[root@dhcp-189-249 lvm-tests]# lvs -a -o +devices
>  LV                 VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert Devices                                
>  lv_opt             vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00         lv_opt_mimage_0(0),lv_opt_mimage_1(0)  
>  [lv_opt_mimage_0]  vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(0)                           
>  [lv_opt_mimage_1]  vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(5000)                        
>  [lv_opt_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13001)                       
>  lv_root            vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00         lv_root_mimage_0(0),lv_root_mimage_1(0)
>  [lv_root_mimage_0] vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(5000)                        
>  [lv_root_mimage_1] vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(0)                           
>  [lv_root_mlog]     vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13000)                       
>  lv_swap            vg_alain     -wi-ao--  1.95g                                                     /dev/sda2(10000)                       
>  lv_var             vg_alain     mwi-aom-  9.77g                         lv_var_mlog  100.00         lv_var_mimage_0(0),lv_var_mimage_1(0)  
>  [lv_var_mimage_0]  vg_alain     iwi-aom-  9.77g                                                     /dev/sda2(10500)                       
>  [lv_var_mimage_1]  vg_alain     iwi-aom-  9.77g                                                     /dev/sdb2(10000)                       
>  [lv_var_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13002)                       
>  lv_home            vg_alain_app mwi-aom- 19.53g                         lv_home_mlog 100.00         lv_home_mimage_0(0),lv_home_mimage_1(0)
>  [lv_home_mimage_0] vg_alain_app iwi-aom- 19.53g                                                     /dev/sda3(0)                           
>  [lv_home_mimage_1] vg_alain_app iwi-aom- 19.53g                                                     /dev/sdb3(2500)                        
>  [lv_home_mlog]     vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7501)                        
>  lv_oam             vg_alain_app mwi-aom-  9.77g                         lv_oam_mlog  100.00         lv_oam_mimage_0(0),lv_oam_mimage_1(0)  
>  [lv_oam_mimage_0]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sda3(5000)                        
>  [lv_oam_mimage_1]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sdb3(0)                           
>  [lv_oam_mlog]      vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7500)           
>             
>
>Adding var breaks, will try again and get a dump, now that we know where to focus
>Then look at dump
>
>vgreduce hangs, find that task in dump, check dentry for what its hanging on
>
>Where is is hanging, 
>
>-----------------------------------------------------------------------------------------------
>crash> bt -f
>PID: 10589  TASK: ffff88007d2d5500  CPU: 0   COMMAND: "vgreduce"
> #0 [ffff88007ce53a38] schedule at ffffffff814fd7e2
>    ffff88007ce53a40: 0000000000000086 0000000000000000 
>    ffff88007ce53a50: ffff88007faf3c40 ffff88007ce53ac8 
>    ffff88007ce53a60: ffffffff8116257b 0000000000000000 
>    ffff88007ce53a70: 0000000000000380 ffff88007d2d5ab8 
>    ffff88007ce53a80: ffff88007ce53fd8 000000000000fb88 
>    ffff88007ce53a90: ffff88007d2d5ab8 ffffffff81a8d020 
>    ffff88007ce53aa0: ffff88007d2d5500 0000000000000002 
>    ffff88007ce53ab0: 0000000000000000 ffff88007ce53af8 
>    ffff88007ce53ac0: 0000000000000000 ffff88007ce53ae8 
>    ffff88007ce53ad0: ffff880002040ef0 ffff88007acc5cf8 
>    ffff88007ce53ae0: ffff88007ce53b78 ffff8800770293c0 
>    ffff88007ce53af0: 0000000000000000 ffff88007ce53bd8 
>    ffff88007ce53b00: ffffffffa00d70ad 
> #1 [ffff88007ce53b00] do_get_write_access at ffffffffa00d70ad [jbd2]
>    ffff88007ce53b08: ffff88007ce53b88 ffff88007ce53b68 
>    ffff88007ce53b18: ffff88007c311f00 ffff88007afd2748 
>    ffff88007ce53b28: ffff880076d717b0 ffff88007d2d5500 
>    ffff88007ce53b38: ffff880076d8b3f0 ffff88007ce53b78 
>    ffff88007ce53b48: ffff88007ce53b90 0000000076db0200 
>    ffff88007ce53b58: ffff880037641000 ffff880002040ef0 
>    ffff88007ce53b68: ffff88007c311f00 0000000000000018 
>    ffff88007ce53b78: 0000000000000000 ffff88007d2d5500 
>    ffff88007ce53b88: ffffffff81092110 ffff880002040ef8 
>    ffff88007ce53b98: ffff880002040ef8 ffffffff812145c0 
>    ffff88007ce53ba8: 0000000000000000 ffff88007acc5cf8 
>    ffff88007ce53bb8: ffff880076d8b3f0 ffff88007c311f00 
>    ffff88007ce53bc8: 0000000000000000 0000000000001885 
>    ffff88007ce53bd8: ffff88007ce53bf8 ffffffffa00d7481 
> #2 [ffff88007ce53be0] jbd2_journal_get_write_access at ffffffffa00d7481 [jbd2]
>    ffff88007ce53be8: ffffffffa013d067 ffff880076d8b3f0 
>    ffff88007ce53bf8: ffff88007ce53c38 ffffffffa012d738 
> #3 [ffff88007ce53c00] __ext4_journal_get_write_access at ffffffffa012d738 [ext4]
>    ffff88007ce53c08: ffff88007ce53c38 ffff880037c43400 
>    ffff88007ce53c18: 0000000000000000 ffff880037c43400 
>    ffff88007ce53c28: ffff88007c311f00 0000000000000000 
>    ffff88007ce53c38: ffff88007ce53d28 ffffffffa0103ab8 
> #4 [ffff88007ce53c40] ext4_new_inode at ffffffffa0103ab8 [ext4]
>    ffff88007ce53c48: ffff880076d8b3f0 ffff88007ce53c90 
>    ffff88007ce53c58: ffff880037641068 ffff88007ae58800 
>    ffff88007ce53c68: ffff88007ad894c0 000081b637097cc0 
>    ffff88007ce53c78: ffff880037dae800 000080008121dd1c 
>    ffff88007ce53c88: ffff8800021ac640 000000000000009d 
>    ffff88007ce53c98: ffff88007ae588c0 0000000000000246 
>    ffff88007ce53ca8: 0000000000000001 ffff880076d8b3f0 
>    ffff88007ce53cb8: ffff880076d8b3f0 ffff88007d2d5500 
>    ffff88007ce53cc8: 000000000000002a ffff88007ad01ef0 
>    ffff88007ce53cd8: ffff88007ce53d18 ffffffffa00d63c5 
>    ffff88007ce53ce8: ffff88007c307dc8 0000003200000000 
>    ffff88007ce53cf8: ffff88007ad01ec0 ffff880076d8b3f0 
>    ffff88007ce53d08: ffff88007ae588c0 ffff88007ad01ec0 
>    ffff88007ce53d18: 00000000000081b6 ffff88007ad01ef0 
>    ffff88007ce53d28: ffff88007ce53d88 ffffffffa01120e0 
> #5 [ffff88007ce53d30] ext4_create at ffffffffa01120e0 [ext4]
>    ffff88007ce53d38: ffff88007ce53d68 ffff88007ad01ec0 
>    ffff88007ce53d48: ffff88007ae588c0 00000000000081b6 
>    ffff88007ce53d58: ffff88007ce53e28 ffff88007ad01ec0 
>    ffff88007ce53d68: ffff88007ae588c0 00000000000081b6 
>    ffff88007ce53d78: ffff88007ce53e28 ffff88007b7da840 
>    ffff88007ce53d88: ffff88007ce53dc8 ffffffff811889b4 
> #6 [ffff88007ce53d90] vfs_create at ffffffff811889b4
>    ffff88007ce53d98: ffff8800000001b6 ffff88007ce53e28 
>    ffff88007ce53da8: ffff88007ce53dc8 00000000000084c1 
>    ffff88007ce53db8: ffff88007b7da840 000000000000002a 
>    ffff88007ce53dc8: ffff88007ce53f18 ffffffff8118c56f 
> #7 [ffff88007ce53dd0] do_filp_open at ffffffff8118c56f
>    ffff88007ce53dd8: 00000000007fffff 00000001ffafffac 
>    ffff88007ce53de8: ffff880000000000 ffffffff81188331 
>    ffff88007ce53df8: ffff88007d2a9000 ffff88007ce53ed8 
>    ffff88007ce53e08: 0000000000000017 0000008037465a80 
>    ffff88007ce53e18: 00000000000001b6 ffff88007d2a9000 
>    ffff88007ce53e28: ffff880037975d80 ffff88007b7da840 
>    ffff88007ce53e38: 0000002f24d677e1 ffff88007d2a9011 
>    ffff88007ce53e48: ffff880037975d80 ffff88007c338300 
>    ffff88007ce53e58: 0000000000000700 0000000000000000 
>    ffff88007ce53e68: 0000000000000000 0000000000000000 
>    ffff88007ce53e78: 0000000000000000 0000000000000086 
>    ffff88007ce53e88: ffff880037465b00 ffff88000c411960 
>    ffff88007ce53e98: ffffffff81aa5700 ffff88007cee7d80 
>    ffff88007ce53ea8: ffff880037ee1d00 000001b6000084c2 
>    ffff88007ce53eb8: ffff88007cce9d80 0000000000000000 
>    ffff88007ce53ec8: ffff88007ce53f18 ffffffff81198182 
>    ffff88007ce53ed8: ffff880037975d80 ffff88007ad01ec0 
>    ffff88007ce53ee8: ffff88007ce53f18 0000000000000006 
>    ffff88007ce53ef8: 00000000ffffff9c ffff88007d2a9000 
>    ffff88007ce53f08: 00000000000084c1 00000000000001b6 
>    ffff88007ce53f18: ffff88007ce53f68 ffffffff81178639 
> #8 [ffff88007ce53f20] do_sys_open at ffffffff81178639
>    ffff88007ce53f28: ffff880037adce48 ffff880037adce48 
>    ffff88007ce53f38: ffff88007ce53f78 000000005a6e8978 
>    ffff88007ce53f48: 00007fff5089b5dc 00007fff5089b3c0 
>    ffff88007ce53f58: 0000000000001000 00000000021f1bc0 
>    ffff88007ce53f68: ffff88007ce53f78 ffffffff81178750 
> #9 [ffff88007ce53f70] sys_open at ffffffff81178750
>    ffff88007ce53f78: 00007fff5089c5e0 ffffffff8100b0f2 
>#10 [ffff88007ce53f80] system_call_fastpath at ffffffff8100b0f2
>    RIP: 00007f44087f9170  RSP: 00007fff5089ae00  RFLAGS: 00010202
>    RAX: 0000000000000002  RBX: ffffffff8100b0f2  RCX: 0000000002437160
>    RDX: 00000000000001b6  RSI: 00000000000004c1  RDI: 00007fff5089c5e0
>    RBP: 00007fff5089c5e0   R8: 0000000000000000   R9: 0000000000000000
>    R10: 00000000ffffffe7  R11: 0000000000000246  R12: ffffffff81178750
>    R13: ffff88007ce53f78  R14: 00000000021f1bc0  R15: 0000000000001000
>    ORIG_RAX: 0000000000000002  CS: 0033  SS: 002b
>
>
>crash> struct dentry ffff88007b7da840 
>struct dentry {
>  d_count = {
>    counter = 19
>  }, 
>  d_flags = 8, 
>  d_lock = {
>    raw_lock = {
>      slock = 0
>    }
>  }, 
>  d_mounted = 1048579, 
>  d_inode = 0xffff88007ae588c0, 
>  d_hash = {
>    next = 0x0, 
>    pprev = 0xffffc900000bb3e0
>  }, 
>  d_parent = 0xffff880076d499c0, 
>  d_name = {
>    hash = 3822982300, 
>    len = 7, 
>    name = 0xffff88007b7da8e0 "archive"
>  }, 
>  d_lru = {
>    next = 0xffff88007b7d2900, 
>    prev = 0xffff88007b7c83c0
>  }, 
>  d_u = {
>    d_child = {
>      next = 0xffff88007addd1d0, 
>      prev = 0xffff880076d49a20
>    }, 
>    d_rcu = {
>      next = 0xffff88007addd1d0, 
>      func = 0xffff880076d49a20
>    }
>  }, 
>  d_subdirs = {
>    next = 0xffff88007ad01f10, 
>    prev = 0xffff88007b7c83d0
>  }, 
>  d_alias = {
>    next = 0xffff88007ae588f0, 
>    prev = 0xffff88007ae588f0
>  }, 
>  d_time = 18442521966238760961, 
>  d_op = 0x0, 
>  d_sb = 0xffff880037c43400, 
>  d_fsdata = 0x0, 
>  d_iname = "archive\000in:aux\000\000\034\000\000\000\001\000\021\000\200\000\000\000\000\000\000"
>}
>
>Hangs trying to write in /etc/lvm/archve
>
>So change lvm.conf and move all access to /boot/etc/lvm
>
>This patch
>
>Apply the patch here or modify /etc/lvm/lvm.conf
>
>[loberman@dhcp-191-51 lvm]$ diff -Nur a/lvm.conf b/lvm.conf.boot
>--- a/lvm.conf  2013-01-18 10:42:24.216174859 -0500
>+++ b/lvm.conf.boot     2013-01-20 13:40:21.129860813 -0500
>@@ -81,12 +81,12 @@
> 
>     # The results of the filtering are cached on disk to avoid
>     # rescanning dud devices (which can take a very long time).
>-    # By default this cache is stored in the /etc/lvm/cache directory
>+    # By default this cache is stored in the /boot/etc/lvm/cache directory
>     # in a file called '.cache'.
>     # It is safe to delete the contents: the tools regenerate it.
>     # (The old setting 'cache' is still respected if neither of
>     # these new ones is present.)
>-    cache_dir = "/etc/lvm/cache"
>+    cache_dir = "/boot/etc/lvm/cache"
>     cache_file_prefix = ""
> 
>     # You can turn off writing this cache file by setting this to 0.
>@@ -285,7 +285,7 @@
> 
>     # Where shall we keep it ?
>     # Remember to back up this directory regularly!
>-    backup_dir = "/etc/lvm/backup"
>+    backup_dir = "/boot/etc/lvm/backup"
> 
>     # Should we maintain an archive of old metadata configurations.
>     # Use 1 for Yes; 0 for No.
>@@ -294,7 +294,7 @@
> 
>     # Where should archived files go ?
>     # Remember to back up this directory regularly!
>-    archive_dir = "/etc/lvm/archive"
>+    archive_dir = "/boot/etc/lvm/archive"
> 
>     # What is the minimum number of archive files you wish to keep ?
>     retain_min = 10
>@@ -388,7 +388,7 @@
> 
>     # Local non-LV directory that holds file-based locks while commands are
>     # in progress.  A directory like /tmp that may get wiped on reboot is OK.
>-    locking_dir = "/var/lock/lvm"
>+    locking_dir = "/boot/var/lock/lvm"
> 
>     # Whenever there are competing read-only and read-write access requests for
>     # a volume group's metadata, instead of always granting the read-only
>@@ -615,7 +615,7 @@
>     #            policy acts like "remove" if no suitable device and space can
>     #            be allocated for the replacement.
> 
>-    mirror_log_fault_policy = "allocate"
>+    mirror_log_fault_policy = "remove"
>     mirror_image_fault_policy = "remove"
> 
>     # 'snapshot_autoextend_threshold' and 'snapshot_autoextend_percent' define
>@@ -744,7 +744,7 @@
>     # you are absolutely sure you know what you are doing! Use
>     # the supplied toolset to make changes (e.g. vgcfgrestore).
> 
>-    # dirs = [ "/etc/lvm/metadata", "/mnt/disk2/lvm/metadata2" ]
>+    # dirs = [ "/boot/etc/lvm/metadata", "/mnt/disk2/lvm/metadata2" ]
> #}
>
>mkdir -p /boot/etc/lvm
>cd /etc/lvm
>cp -R * /boot/etc/lvm
>dracut -f
>reboot and test
>
>[root@dhcp-189-9 ~]# lvs -a -o +devices
>  LV                 VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert Devices                                
>  lv_opt             vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00         lv_opt_mimage_0(0),lv_opt_mimage_1(0)  
>  [lv_opt_mimage_0]  vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(0)                           
>  [lv_opt_mimage_1]  vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(5000)                        
>  [lv_opt_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13001)                       
>  lv_root            vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00         lv_root_mimage_0(0),lv_root_mimage_1(0)
>  [lv_root_mimage_0] vg_alain     iwi-aom- 19.53g                                                     /dev/sda2(5000)                        
>  [lv_root_mimage_1] vg_alain     iwi-aom- 19.53g                                                     /dev/sdb2(0)                           
>  [lv_root_mlog]     vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13000)                       
>  lv_swap            vg_alain     -wi-ao--  1.95g                                                     /dev/sda2(10000)                       
>  lv_var             vg_alain     mwi-aom-  9.77g                         lv_var_mlog  100.00         lv_var_mimage_0(0),lv_var_mimage_1(0)  
>  [lv_var_mimage_0]  vg_alain     iwi-aom-  9.77g                                                     /dev/sda2(10500)                       
>  [lv_var_mimage_1]  vg_alain     iwi-aom-  9.77g                                                     /dev/sdb2(10000)                       
>  [lv_var_mlog]      vg_alain     lwa-aom-  4.00m                                                     /dev/sda2(13002)                       
>  lv_home            vg_alain_app mwi-aom- 19.53g                         lv_home_mlog 100.00         lv_home_mimage_0(0),lv_home_mimage_1(0)
>  [lv_home_mimage_0] vg_alain_app iwi-aom- 19.53g                                                     /dev/sda3(0)                           
>  [lv_home_mimage_1] vg_alain_app iwi-aom- 19.53g                                                     /dev/sdb3(2500)                        
>  [lv_home_mlog]     vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7501)                        
>  lv_oam             vg_alain_app mwi-aom-  9.77g                         lv_oam_mlog  100.00         lv_oam_mimage_0(0),lv_oam_mimage_1(0)  
>  [lv_oam_mimage_0]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sda3(5000)                        
>  [lv_oam_mimage_1]  vg_alain_app iwi-aom-  9.77g                                                     /dev/sdb3(0)                           
>  [lv_oam_mlog]      vg_alain_app lwa-aom-  4.00m                                                     /dev/sda3(7500)                        
>
>remove or disable mirror device (in this case sdb)
>
>lvs and vgdisplay no longer hang but have to use vgreduce for manual intervention.
>If you dont do this and reboot, system will stop unable to access volumes or will panic.
>
>[root@dhcp-189-9 ~]# lvs
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid 7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF.
>  Couldn't find device with uuid QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH.
>  LV      VG           Attr     LSize  Pool Origin Data%  Move Log          Copy%  Convert
>  lv_opt  vg_alain     mwi-aom- 19.53g                         lv_opt_mlog  100.00        
>  lv_root vg_alain     mwi-aom- 19.53g                         lv_root_mlog 100.00        
>  lv_swap vg_alain     -wi-ao--  1.95g                                                    
>  lv_var  vg_alain     -wi-ao--  9.77g                                                    
>  lv_home vg_alain_app mwi-aom- 19.53g                         lv_home_mlog 100.00        
>  lv_oam  vg_alain_app mwi-aom-  9.77g                         lv_oam_mlog  100.00        
>[root@dhcp-189-9 ~]# vgdisplay
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid 7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF.
>  --- Volume group ---
>  VG Name               vg_alain_app
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  8
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                2
>  Open LV               2
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               78.12 GiB
>  PE Size               4.00 MiB
>  Total PE              19998
>  Alloc PE / Size       15002 / 58.60 GiB
>  Free  PE / Size       4996 / 19.52 GiB
>  VG UUID               Ijnhwf-VdGk-kMJU-Plxl-orcd-ois1-JTWnCz
>   
>  Couldn't find device with uuid QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH.
>  --- Volume group ---
>  VG Name               vg_alain
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  17
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                4
>  Open LV               4
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               117.18 GiB
>  PE Size               4.00 MiB
>  Total PE              29998
>  Alloc PE / Size       23002 / 89.85 GiB
>  Free  PE / Size       6996 / 27.33 GiB
>  VG UUID               TbtVxL-p0zk-yhX0-v1po-0CuP-LtxK-7U7y3f
>   
>[root@dhcp-189-9 ~]# vgdisplay -v
>    Finding all volume groups
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    Finding volume group "vg_alain_app"
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  Couldn't find device with uuid 7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF.
>    There are 1 physical volumes missing.
>  --- Volume group ---
>  VG Name               vg_alain_app
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  8
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                2
>  Open LV               2
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               78.12 GiB
>  PE Size               4.00 MiB
>  Total PE              19998
>  Alloc PE / Size       15002 / 58.60 GiB
>  Free  PE / Size       4996 / 19.52 GiB
>  VG UUID               Ijnhwf-VdGk-kMJU-Plxl-orcd-ois1-JTWnCz
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_home
>  LV Name                lv_home
>  VG Name                vg_alain_app
>  LV UUID                rUde9I-AAvO-1bLF-mpqM-52hX-nrSX-T7Vsxa
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:32 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Mirrored volumes       2
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:8
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_oam
>  LV Name                lv_oam
>  VG Name                vg_alain_app
>  LV UUID                BWRJBP-1XSo-Mh5z-LBFl-NJ5X-c2p2-BpeFdp
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:33 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Mirrored volumes       2
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:12
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda3     
>  PV UUID               gdCaVp-wBWa-NgaC-J4v6-8i6I-Q3wE-UVzD5J
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 2497
>   
>  PV Name               unknown device     
>  PV UUID               7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 2499
>   
>    Finding volume group "vg_alain"
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  Couldn't find device with uuid QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH.
>    There are 1 physical volumes missing.
>  --- Volume group ---
>  VG Name               vg_alain
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  22
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                4
>  Open LV               4
>  Max PV                0
>  Cur PV                2
>  Act PV                1
>  VG Size               117.18 GiB
>  PE Size               4.00 MiB
>  Total PE              29998
>  Alloc PE / Size       18001 / 70.32 GiB
>  Free  PE / Size       11997 / 46.86 GiB
>  VG UUID               TbtVxL-p0zk-yhX0-v1po-0CuP-LtxK-7U7y3f
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_opt
>  LV Name                lv_opt
>  VG Name                vg_alain
>  LV UUID                Vb73IK-H2Nh-7nZU-zH2x-tXcN-r7dv-Fb7Ss3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:26 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Mirrored volumes       2
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:16
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_root
>  LV Name                lv_root
>  VG Name                vg_alain
>  LV UUID                6ZPvoM-dHj3-P6Qd-WesB-VaSS-mOg2-xShaO3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:29 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:3
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_swap
>  LV Name                lv_swap
>  VG Name                vg_alain
>  LV UUID                yYebAk-sbn3-nLC7-VRcm-44Rc-3S3s-vuYbSZ
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:30 -0500
>  LV Status              available
>  # open                 1
>  LV Size                1.95 GiB
>  Current LE             500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:4
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_var
>  LV Name                lv_var
>  VG Name                vg_alain
>  LV UUID                dcSJaM-Gkjy-scbL-fL93-jdgc-cNtO-wrl9q4
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:31 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:20
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda2     
>  PV UUID               707g3M-jZh4-jCQR-GSwO-NOlA-Fnj5-z2Vz25
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 1998
>   
>  PV Name               unknown device     
>  PV UUID               QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 9999
>   
>    Archiving volume group "vg_alain" metadata (seqno 17).
>    Archiving volume group "vg_alain" metadata (seqno 22).
>    Creating volume group backup "/boot/etc/lvm/backup/vg_alain" (seqno 22).
>
>[root@dhcp-189-9 ~]# vgreduce --removemissing --force vg_alain
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid QpvM5R-g4pB-tTIR-ohKd-b041-Xagd-e2j7HH.
>  Wrote out consistent volume group vg_alain
>[root@dhcp-189-9 ~]# vgreduce --removemissing --force vg_alain_app
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>  Couldn't find device with uuid 7GJHq6-nRHN-J3XO-aYkV-6YDn-lxRD-VZxPeF.
>  Wrote out consistent volume group vg_alain_app
>
>[root@dhcp-189-9 ~]# vgdisplay -v
>    Finding all volume groups
>  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111535104: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 118111592448: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524222464: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 524279808: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb1: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914494464: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 62914551808: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb2: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb2: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41942974464: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 41943031808: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>  /dev/sdb3: read failed after 0 of 4096 at 4096: Input/output error
>    /dev/sdb3: read failed after 0 of 4096 at 0: Input/output error
>    Finding volume group "vg_alain_app"
>  --- Volume group ---
>  VG Name               vg_alain_app
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  19
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                2
>  Open LV               2
>  Max PV                0
>  Cur PV                1
>  Act PV                1
>  VG Size               39.06 GiB
>  PE Size               4.00 MiB
>  Total PE              9999
>  Alloc PE / Size       7500 / 29.30 GiB
>  Free  PE / Size       2499 / 9.76 GiB
>  VG UUID               Ijnhwf-VdGk-kMJU-Plxl-orcd-ois1-JTWnCz
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_home
>  LV Name                lv_home
>  VG Name                vg_alain_app
>  LV UUID                rUde9I-AAvO-1bLF-mpqM-52hX-nrSX-T7Vsxa
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:32 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:8
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain_app/lv_oam
>  LV Name                lv_oam
>  VG Name                vg_alain_app
>  LV UUID                BWRJBP-1XSo-Mh5z-LBFl-NJ5X-c2p2-BpeFdp
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:33 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:12
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda3     
>  PV UUID               gdCaVp-wBWa-NgaC-J4v6-8i6I-Q3wE-UVzD5J
>  PV Status             allocatable
>  Total PE / Free PE    9999 / 2499
>   
>    Finding volume group "vg_alain"
>  --- Volume group ---
>  VG Name               vg_alain
>  System ID             
>  Format                lvm2
>  Metadata Areas        1
>  Metadata Sequence No  29
>  VG Access             read/write
>  VG Status             resizable
>  MAX LV                0
>  Cur LV                4
>  Open LV               4
>  Max PV                0
>  Cur PV                1
>  Act PV                1
>  VG Size               58.59 GiB
>  PE Size               4.00 MiB
>  Total PE              14999
>  Alloc PE / Size       13000 / 50.78 GiB
>  Free  PE / Size       1999 / 7.81 GiB
>  VG UUID               TbtVxL-p0zk-yhX0-v1po-0CuP-LtxK-7U7y3f
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_opt
>  LV Name                lv_opt
>  VG Name                vg_alain
>  LV UUID                Vb73IK-H2Nh-7nZU-zH2x-tXcN-r7dv-Fb7Ss3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:26 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:16
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_root
>  LV Name                lv_root
>  VG Name                vg_alain
>  LV UUID                6ZPvoM-dHj3-P6Qd-WesB-VaSS-mOg2-xShaO3
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:29 -0500
>  LV Status              available
>  # open                 1
>  LV Size                19.53 GiB
>  Current LE             5000
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:3
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_swap
>  LV Name                lv_swap
>  VG Name                vg_alain
>  LV UUID                yYebAk-sbn3-nLC7-VRcm-44Rc-3S3s-vuYbSZ
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:30 -0500
>  LV Status              available
>  # open                 1
>  LV Size                1.95 GiB
>  Current LE             500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:4
>   
>  --- Logical volume ---
>  LV Path                /dev/vg_alain/lv_var
>  LV Name                lv_var
>  VG Name                vg_alain
>  LV UUID                dcSJaM-Gkjy-scbL-fL93-jdgc-cNtO-wrl9q4
>  LV Write Access        read/write
>  LV Creation host, time localhost.localdomain, 2013-01-16 09:12:31 -0500
>  LV Status              available
>  # open                 1
>  LV Size                9.77 GiB
>  Current LE             2500
>  Segments               1
>  Allocation             inherit
>  Read ahead sectors     auto
>  - currently set to     256
>  Block device           253:20
>   
>  --- Physical volumes ---
>  PV Name               /dev/sda2     
>  PV UUID               707g3M-jZh4-jCQR-GSwO-NOlA-Fnj5-z2Vz25
>  PV Status             allocatable
>  Total PE / Free PE    14999 / 1999
>
>Reboot and test
>Looks fine, able to add back and re-mirror

Actions: View

Attachments on bug 905063: 915664