Bug 929193

Have you tried to use:

vgextend --restoremissing vgname  pvnames...

I know that using vgextend/vgreduce is an option to change their state, but I have not tried to do so. They should not have been marked as a-m when using mirror_segtype_default=mirror. 

The activity of marking PVs as missing and then reinstating them via vgextend/vgreduce was expected to be used with raid1 segtype, but not with old mirror segtype afaik.

If I am correct in my assumption:

1) Why are PVs marked missing when they are back in ordinary mirrors?

2) Why don't they come back on their own, but do back in a bunch once the last PV has returned without any vgextend intevention in-between (there were lvm commands executed in the meantime, so it is not a timing issue)?


If I am not correct with my segtype=mirror assumption:

Is this marking of PVs as 'a-m' now an expected behavior for the returning (running) PVs of ordinary mirror segtype? 

If so, then:

a) it has to be documented and 
b) PVs should never lose the a-m flags on a whim, for as long as they are part of the same VG they were in when they went offline (otherwise what would be the point of such an unstable a-m flag).

Is it using the 'missing' devices when they come back?  (It shouldn't in case something was changed on them while disconnected.  Unless you run --restoremissing to confirm that nothing was changed on them, the system has to do a full resync.)

Well it just removed the a-m flags from the returned devices (as you can see in my opening comment), without me doing any --restoremissing

They sat there as a-m:

  /dev/sdb1                                                ---       0      0 
  /dev/sdc1                                                ---       0      0 
  /dev/sdd1                                                ---       0      0 
  /dev/sde1                                                ---       0      0 
  /dev/sdf1                      revolution_9         lvm2 a-m   93.12g 93.12g
  /dev/sdg1                      revolution_9         lvm2 a-m   93.12g 93.12g
  /dev/sdi1                      revolution_9         lvm2 a-m   93.12g 93.12g
  /dev/sdj1                      revolution_9         lvm2 a--   93.12g 91.12g
  /dev/sdk1                      revolution_9         lvm2 a-m   93.12g 93.12g

(note that /dev/sdh1 is not there, since it is offline) 

The devices are all online, LVM sees their names, and they are a-m.

Now I return /dev/sdh1 from the dead (only that, no other commands done) and do a vgs:

[root@tardis-01-port1 ~]# echo "running" >/sys/block/sdh/device/state
[root@tardis-01-port1 ~]# vgs
  WARNING: Inconsistent metadata found for VG revolution_9 - updating to use version 33
  Missing device /dev/sdk1 reappeared, updating metadata for VG revolution_9 to version 33.
  Missing device /dev/sdi1 reappeared, updating metadata for VG revolution_9 to version 33.
  Missing device /dev/sdh1 reappeared, updating metadata for VG revolution_9 to version 33.
  Missing device /dev/sdg1 reappeared, updating metadata for VG revolution_9 to version 33.
  Missing device /dev/sdf1 reappeared, updating metadata for VG revolution_9 to version 33.
  VG                   #PV #LV #SN Attr   VSize   VFree  
  revolution_9           6   1   0 wz--n- 558.75g 556.75g
  rhel_tardis-01-port1   1   3   0 wz--n- 278.88g      0

The segment type has nothing to do with treatment of missing PVs. Even if you don't have mirrors at all, same thing will happen, once you write to a VG, PVs will be marked as missing. Unless you write to the VG (whether due to recovery or unrelated action) the missing flag won't be cleared.

As for PVs coming back on a whim, I was always opposed to that, but code to sometimes do that has been added to LVM. It's not entirely correct but will work more often than not. It probably won't lose data as long as metadata backups are working correctly.

Overall, I don't think this is a problem. If you think some behaviour needs to change in the code, please re-open the bug, ideally with an example of a run where you think the current logic fails. Two kinds of failures could happen:
- it is entirely clear that the PVs in question could not change while it was
  missing, but the code did not detect this
- the code automatically re-integrated a PV which was modified externally and
  possibly wiped the changed metadata in the process

Please do keep in mind that when an automated mirror recovery kicks in (whether hotspare or downconvert), this also means that the metadata has diverged and we currently do not treat these cases as safe to reclaim (fails the "entirely clear" criterion from point 1).

(PS: Using a chain-(crypto)hashing scheme for metadata versioning would presumably allow us to establish clear relationships between random snapshots of the same VG. Whether this is viable is an exercise for the future.)