Created attachment 495151 [details]
logs.

Description of problem:
On spm fail-over vgchange is executed -  addtag with new MDT_SHA_CKSUM value and deltag with non-existent cached old MDT_SHA_CKSUM which cause checksum mismatch.

on the first host:

dd76327f-141f-4ec7-8820-116536d896b0::DEBUG::2011-04-27 10:12:57,268::lvm::358::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_na
mes = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 filter = [ \\"a%/dev/mapper/1DD1|/dev/mapper/1DNAORI1|/dev/mapper/1DNAORI2|/dev/mapper/1DNAORI3|/dev/m
apper/1David|/dev/mapper/1David2|/dev/mapper/1David3|/dev/mapper/1David4|/dev/mapper/1David5|/dev/mapper/1David6|/dev/mapper/1David7|/dev/mapper/1David8|/dev/mapper/1David9|/dev
/mapper/1LIBVIRT|/dev/mapper/1LIBVIRT2|/dev/mapper/1LIBVIRT3|/dev/mapper/1size%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  back
up {  retain_min = 50  retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT_POOL_SPM_LVER=10 --deltag MDT__SHA_CKSUM=80a1d3d9853b52513e91f2c6566376c3e182e596 --addtag MDT
_POOL_SPM_ID=1 --addtag MDT__SHA_CKSUM=7791c76f93e5562395cac02d84961d94643b193c --addtag MDT_POOL_SPM_LVER=11 c9f2fca8-35da-4d4c-ac73-9bd7748042e4' (cwd None)

on the second host:

b8c74cd8-5df4-41a3-a1d4-939bf9550476::DEBUG::2011-04-27 10:21:02,131::lvm::358::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_na
mes = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 filter = [ \\"a%/dev/mapper/1DD1|/dev/mapper/1DNAORI1|/dev/mapper/1DNAORI2|/dev/mapper/1DNAORI3|/dev/m
apper/1David|/dev/mapper/1David2|/dev/mapper/1David3|/dev/mapper/1David4|/dev/mapper/1David5|/dev/mapper/1David6|/dev/mapper/1David7|/dev/mapper/1David8|/dev/mapper/1David9|/dev
/mapper/1LIBVIRT|/dev/mapper/1LIBVIRT2|/dev/mapper/1LIBVIRT3|/dev/mapper/1size%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  back
up {  retain_min = 50  retain_days = 0 } " --deltag MDT_POOL_SPM_LVER=10 --deltag MDT__SHA_CKSUM=80a1d3d9853b52513e91f2c6566376c3e182e596 --addtag MDT__SHA_CKSUM=6024ad44f543655
d9654aa06c6ac7f394895234b --addtag MDT_POOL_SPM_LVER=11 c9f2fca8-35da-4d4c-ac73-9bd7748042e4' (cwd None)

....

Thread-22::ERROR::2011-04-27 10:21:17,351::task::855::TaskManager.Task::(_setError) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 863, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/hsm.py", line 564, in public_connectStoragePool
    return pool.connect(hostID, scsiKey, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 398, in connect
    self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1201, in getMasterDomain
    self.masterDomain = self.verifyMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1227, in verifyMasterDomain
    if not domain.isMaster():
  File "/usr/share/vdsm/storage/sd.py", line 658, in isMaster
    return self.getMetaParam(DMDK_ROLE).capitalize() == MASTER_DOMAIN
  File "/usr/share/vdsm/storage/sd.py", line 621, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 175, in __getitem__
    with self._accessWrapper():
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 125, in _accessWrapper
    self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 245, in refresh
    raise se.MetaDataSealIsBroken(declaredChecksum, computedChecksum)
MetaDataSealIsBroken: Meta Data seal is broken (checksum mismatch): u'cksum = 6024ad44f543655d9654aa06c6ac7f394895234b, computed_cksum = 7791c76f93e5562395cac02d84961d94643b193c

Version-Release number of selected component (if applicable):
vdsm-4.9-62.el6.x86_64
lvm2-2.02.83-3.el6.x86_64

Additional info:
attached vdsm logs from both hosts.