================================================================================
Iteration 0.3 started at Tue May  2 14:20:49 CDT 2017
================================================================================
  WARNING: Not using lvmetad because a repair command was run.
Scenario kill_three_synced_raid10_3legs: Kill three legs (none of which share the same stripe leg) of synced 3 leg raid10 volume(s)

********* RAID hash info for this scenario *********
* names:              synced_three_raid10_3legs_1
* sync:               1
* type:               raid10
* -m |-i value:       3
* leg devices:        /dev/sdh1 /dev/sdg1 /dev/sdc1 /dev/sdf1 /dev/sda1 /dev/sde1
* spanned legs:       0
* manual repair:      0
* no MDA devices:     
* failpv(s):          /dev/sdh1 /dev/sdc1 /dev/sda1
* failnode(s):        host-073
* lvmetad:            1
* raid fault policy:  allocate
******************************************************

Creating raids(s) on host-073...
host-073: lvcreate  --type raid10 -i 3 -n synced_three_raid10_3legs_1 -L 500M black_bird /dev/sdh1:0-2400 /dev/sdg1:0-2400 /dev/sdc1:0-2400 /dev/sdf1:0-2400 /dev/sda1:0-2400 /dev/sde1:0-2400
  WARNING: Not using lvmetad because a repair command was run.

Current mirror/raid device structure(s):
  WARNING: Not using lvmetad because a repair command was run.
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_three_raid10_3legs_1            rwi-aor--- 504.00m 80.16    synced_three_raid10_3legs_1_rimage_0(0),synced_three_raid10_3legs_1_rimage_1(0),synced_three_raid10_3legs_1_rimage_2(0),synced_three_raid10_3legs_1_rimage_3(0),synced_three_raid10_3legs_1_rimage_4(0),synced_three_raid10_3legs_1_rimage_5(0)
   [synced_three_raid10_3legs_1_rimage_0] Iwi-aor--- 168.00m          /dev/sdh1(1)
   [synced_three_raid10_3legs_1_rimage_1] Iwi-aor--- 168.00m          /dev/sdg1(1)
   [synced_three_raid10_3legs_1_rimage_2] Iwi-aor--- 168.00m          /dev/sdc1(1)
   [synced_three_raid10_3legs_1_rimage_3] Iwi-aor--- 168.00m          /dev/sdf1(1)
   [synced_three_raid10_3legs_1_rimage_4] Iwi-aor--- 168.00m          /dev/sda1(1)
   [synced_three_raid10_3legs_1_rimage_5] Iwi-aor--- 168.00m          /dev/sde1(1)
   [synced_three_raid10_3legs_1_rmeta_0]  ewi-aor---   4.00m          /dev/sdh1(0)
   [synced_three_raid10_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sdg1(0)
   [synced_three_raid10_3legs_1_rmeta_2]  ewi-aor---   4.00m          /dev/sdc1(0)
   [synced_three_raid10_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdf1(0)
   [synced_three_raid10_3legs_1_rmeta_4]  ewi-aor---   4.00m          /dev/sda1(0)
   [synced_three_raid10_3legs_1_rmeta_5]  ewi-aor---   4.00m          /dev/sde1(0)

* NOTE: not enough available devices for allocation fault polices to fully work *
(well technically, since we have 1, some allocation should work)

Waiting until all mirror|raid volumes become fully syncd...
   1/1 mirror(s) are fully synced: ( 100.00% )
Sleeping 15 sec

Creating xfs on top of mirror(s) on host-073...
Mounting mirrored xfs filesystems on host-073...

PV=/dev/sda1
        synced_three_raid10_3legs_1_rimage_4: 1.P
        synced_three_raid10_3legs_1_rmeta_4: 1.P
PV=/dev/sdc1
        synced_three_raid10_3legs_1_rimage_2: 1.P
        synced_three_raid10_3legs_1_rmeta_2: 1.P
PV=/dev/sdh1
        synced_three_raid10_3legs_1_rimage_0: 1.P
        synced_three_raid10_3legs_1_rmeta_0: 1.P

Writing verification files (checkit) to mirror(s) on...
        ---- host-073 ----

Sleeping 15 seconds to get some outsanding I/O locks before the failure 
Verifying files (checkit) on mirror(s) on...
        ---- host-073 ----

Disabling device sdh on host-073rescan device...
Disabling device sdc on host-073rescan device...
Disabling device sda on host-073rescan device...

Getting recovery check start time from /var/log/messages: May  2 14:22
Attempting I/O to cause mirror down conversion(s) on host-073
dd if=/dev/zero of=/mnt/synced_three_raid10_3legs_1/ddfile count=10 bs=4M
10+0 records in
10+0 records out
41943040 bytes (42 MB) copied, 0.0922284 s, 455 MB/s

Verifying current sanity of lvm after the failure

Current mirror/raid device structure(s):
  WARNING: Not using lvmetad because a repair command was run.
  Couldn't find device with uuid 2wqEs7-9v93-0cLU-7emZ-ZSfu-QyLv-dpZ7vN.
  Couldn't find device with uuid gMS2fR-Enm5-YCNL-4dHx-TxUO-S77N-d0QIPr.
  Couldn't find device with uuid MwRsWQ-uEI5-Q7Os-JPyk-B9oj-C3q4-tANjaS.
  LV                                     Attr       LSize   Cpy%Sync Devices
   synced_three_raid10_3legs_1            rwi-aor-p- 504.00m 100.00   synced_three_raid10_3legs_1_rimage_0(0),synced_three_raid10_3legs_1_rimage_1(0),synced_three_raid10_3legs_1_rimage_2(0),synced_three_raid10_3legs_1_rimage_3(0),synced_three_raid10_3legs_1_rimage_4(0),synced_three_raid10_3legs_1_rimage_5(0)
   [synced_three_raid10_3legs_1_rimage_0] Iwi-aor-p- 168.00m          [unknown](1)
   [synced_three_raid10_3legs_1_rimage_1] iwi-aor--- 168.00m          /dev/sdg1(1)
   [synced_three_raid10_3legs_1_rimage_2] Iwi-aor-p- 168.00m          [unknown](1)
   [synced_three_raid10_3legs_1_rimage_3] iwi-aor--- 168.00m          /dev/sdf1(1)
   [synced_three_raid10_3legs_1_rimage_4] iwi-aor--- 168.00m          /dev/sdb1(1)
   [synced_three_raid10_3legs_1_rimage_5] iwi-aor--- 168.00m          /dev/sde1(1)
   [synced_three_raid10_3legs_1_rmeta_0]  ewi-aor-p-   4.00m          [unknown](0)
   [synced_three_raid10_3legs_1_rmeta_1]  ewi-aor---   4.00m          /dev/sdg1(0)
   [synced_three_raid10_3legs_1_rmeta_2]  ewi-aor-p-   4.00m          [unknown](0)
   [synced_three_raid10_3legs_1_rmeta_3]  ewi-aor---   4.00m          /dev/sdf1(0)
   [synced_three_raid10_3legs_1_rmeta_4]  ewi-aor---   4.00m          /dev/sdb1(0)
   [synced_three_raid10_3legs_1_rmeta_5]  ewi-aor---   4.00m          /dev/sde1(0)

Verifying FAILED device /dev/sdh1 is *NOT* in the volume(s)
Verifying FAILED device /dev/sdc1 is *NOT* in the volume(s)
Verifying FAILED device /dev/sda1 is *NOT* in the volume(s)
Verifying IMAGE device /dev/sdg1 *IS* in the volume(s)
Verifying IMAGE device /dev/sdf1 *IS* in the volume(s)
Verifying IMAGE device /dev/sde1 *IS* in the volume(s)
Verify the rimage/rmeta dm devices remain after the failures

Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rimage_4 on: host-073 
Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rmeta_4 on: host-073 
Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rimage_2 on: host-073 
Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rmeta_2 on: host-073 
Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rimage_0 on: host-073 
Checking EXISTENCE and STATE of synced_three_raid10_3legs_1_rmeta_0 on: host-073 

Verify the raid image order is what's expected based on raid fault policy
EXPECTED LEG ORDER: unknown /dev/sdg1 unknown /dev/sdf1 unknown /dev/sde1
ACTUAL LEG ORDER: [unknown] /dev/sdg1 [unknown] /dev/sdf1 /dev/sdb1 /dev/sde1

Verifying files (checkit) on mirror(s) on...
        ---- host-073 ----

Enabling device sdh on host-073  WARNING: Not using lvmetad because a repair command was run.
        Running vgs to make LVM update metadata version if possible (will restore a-m PVs)
        Simple vgs cmd failed after brining sdh back online
        (Possible regression of bug 1412843/1434054)
Enabling device sdc on host-073  WARNING: Not using lvmetad because a repair command was run.
        Running vgs to make LVM update metadata version if possible (will restore a-m PVs)
        Simple vgs cmd failed after brining sdc back online
        (Possible regression of bug 1412843/1434054)
Enabling device sda on host-073  WARNING: Not using lvmetad because a repair command was run.
        Running vgs to make LVM update metadata version if possible (will restore a-m PVs)

-------------------------------------------------------------------------------
Force a vgreduce to clean up the corrupt additional LV
 ( vgreduce --removemissing --force black_bird )
-------------------------------------------------------------------------------

Recreating PVs /dev/sdh1 /dev/sdc1 /dev/sda1 and then extending back into black_bird
host-073 pvcreate /dev/sdh1
host-073 pvcreate /dev/sdc1
host-073 pvcreate /dev/sda1
  WARNING: Not using lvmetad because a repair command was run.
  Can't initialize physical volume "/dev/sda1" of volume group "black_bird" without -ff
recreation of /dev/sda1 failed, must still be in VG
host-073 vgextend black_bird /dev/sdh1
  WARNING: Not using lvmetad because a repair command was run.
host-073 vgextend black_bird /dev/sdc1
  WARNING: Not using lvmetad because a repair command was run.
host-073 vgextend black_bird /dev/sda1
  WARNING: Not using lvmetad because a repair command was run.
  Physical volume '/dev/sda1' is already in volume group 'black_bird'
  Unable to add physical volume '/dev/sda1' to volume group 'black_bird'
extension of /dev/sda1 back into black_bird failed

Checking for leftover '-missing_0_0' or 'unknown devices'
'-missing' devices still exist (normal for partial allocation scenarios, see BUG 825026)
  WARNING: Not using lvmetad because a repair command was run.

Checking for PVs marked as missing (a-m)...
  WARNING: Not using lvmetad because a repair command was run.

Verifying files (checkit) on mirror(s) on...
        ---- host-073 ----

Stopping the io load (collie/xdoio) on mirror(s)
Unmounting xfs and removing mnt point on host-073...

Deactivating and removing raid(s)
May  2 14:26:16 host-073 qarshd[7551]: Running cmdline: lvremove -f /dev/black_bird/synced_three_raid10_3legs_1
This appears to be bug: 1447097


[root@host-073 ~]# lvs -a -o +devices
  LV   VG            Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices       
  root rhel_host-073 -wi-ao----  <6.20g                                                     /dev/vda2(205)
  swap rhel_host-073 -wi-ao---- 820.00m                                                     /dev/vda2(0)  

[root@host-073 ~]# dmsetup ls
black_bird-synced_three_raid10_3legs_1_rmeta_0-missing_0_0      (253:18)
black_bird-synced_three_raid10_3legs_1_rimage_2-missing_0_0     (253:15)
black_bird-synced_three_raid10_3legs_1_rmeta_2-missing_0_0      (253:16)
rhel_host--073-swap     (253:1)
rhel_host--073-root     (253:0)
black_bird-synced_three_raid10_3legs_1_rimage_0-missing_0_0     (253:17)

"lvchange -an" doesn't deactivate any "*-missing_N_0" transient devices yet.

The "missing" devices should not be appearing like this at all - these situations should be handled in a controlled way via the VG metadata.

(In reply to Alasdair Kergon from comment #4)
> The "missing" devices should not be appearing like this at all - these
> situations should be handled in a controlled way via the VG metadata.

We'll do that as the final solution.

With "lvchange --refresh ..." we already have lv_deactivate_any_missing_subdevs() we can call in LV deactivation for the time being.

This is however not handling and transitional states which may happen during conversion/resize operation - so it's really only a temporary hack with no big usability and proper activation-code based solution needs to be found.

(Similar applies to handling caching on failing devices).

Upstream commit 94632eb155a0a1814e69b1baa0516323cbbae648

This patch is unfortunately only somewhat ugly hack.

The core reason here which need major rewrite is - the  raid code is using  'missing' devices for suspend/resume.

Such logic is however unsupported by lvm2 code - so 'missing' devices are leaked during various 'lvconvert' operations that may happen with 'active' LV that was partially activated.

This patch only 'covers' most obvious case where the raid was not i.e. converted and some just run 'lvchange --refresh' on raid with missing devices.

We need proper design to deal with missing devices on 'raid'  where we stop misusing partial activation (where the only supported next state after partial activation is deactivation)

Supporting 'missing' devices for suspend/resume would require rather major rework of dm tree handling so this path is rather not an option for going forward (as there is also further major detach of data in dm table and data committed on disk.

We may try to redesign  handling of missing device.

In practice we need just a single device per VG to handle all missing pieces.

In that case there is easy way to always include deptree for such device and make sure  there are no leaked device when devices reapers..

Also note that ATM there are several bugs in misuse of  --partial activation where several pieces of code needs bugfixing.

Created attachment 1394860 [details]
test_log

Verified. Attaching log with test results.

 
3.10.0-847.el7.x86_64

lvm2-2.02.177-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
lvm2-libs-2.02.177-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
lvm2-cluster-2.02.177-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
cmirror-2.02.177-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
device-mapper-1.02.146-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
device-mapper-libs-1.02.146-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
device-mapper-event-1.02.146-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
device-mapper-event-libs-1.02.146-2.el7    BUILT: Wed Feb  7 17:39:26 CET 2018
device-mapper-persistent-data-0.7.3-3.el7    BUILT: Tue Nov 14 12:07:18 CET 2017

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0853