Bug 700004

Summary: [vdsm][Storage] vgtag metadata double checksum.
Product: Red Hat Enterprise Linux 6 Reporter: David Naori <dnaori>
Component: vdsmAssignee: Saggi Mizrahi <smizrahi>
Status: CLOSED ERRATA QA Contact: Kiril Nesenko <knesenko>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2CC: abaron, bazulay, danken, dfediuck, dnaori, hateya, iheim, knesenko, mgoldboi, sgrinber, ykaul
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: vdsm-4.9-63 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 07:15:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
logs. none

Description David Naori 2011-04-27 08:57:21 UTC
Created attachment 495151 [details]
logs.

Description of problem:
On spm fail-over vgchange is executed -  addtag with new MDT_SHA_CKSUM value and deltag with non-existent cached old MDT_SHA_CKSUM which cause checksum mismatch.

on the first host:

dd76327f-141f-4ec7-8820-116536d896b0::DEBUG::2011-04-27 10:12:57,268::lvm::358::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_na
mes = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 filter = [ \\"a%/dev/mapper/1DD1|/dev/mapper/1DNAORI1|/dev/mapper/1DNAORI2|/dev/mapper/1DNAORI3|/dev/m
apper/1David|/dev/mapper/1David2|/dev/mapper/1David3|/dev/mapper/1David4|/dev/mapper/1David5|/dev/mapper/1David6|/dev/mapper/1David7|/dev/mapper/1David8|/dev/mapper/1David9|/dev
/mapper/1LIBVIRT|/dev/mapper/1LIBVIRT2|/dev/mapper/1LIBVIRT3|/dev/mapper/1size%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  back
up {  retain_min = 50  retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT_POOL_SPM_LVER=10 --deltag MDT__SHA_CKSUM=80a1d3d9853b52513e91f2c6566376c3e182e596 --addtag MDT
_POOL_SPM_ID=1 --addtag MDT__SHA_CKSUM=7791c76f93e5562395cac02d84961d94643b193c --addtag MDT_POOL_SPM_LVER=11 c9f2fca8-35da-4d4c-ac73-9bd7748042e4' (cwd None)

on the second host:

b8c74cd8-5df4-41a3-a1d4-939bf9550476::DEBUG::2011-04-27 10:21:02,131::lvm::358::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_na
mes = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 filter = [ \\"a%/dev/mapper/1DD1|/dev/mapper/1DNAORI1|/dev/mapper/1DNAORI2|/dev/mapper/1DNAORI3|/dev/m
apper/1David|/dev/mapper/1David2|/dev/mapper/1David3|/dev/mapper/1David4|/dev/mapper/1David5|/dev/mapper/1David6|/dev/mapper/1David7|/dev/mapper/1David8|/dev/mapper/1David9|/dev
/mapper/1LIBVIRT|/dev/mapper/1LIBVIRT2|/dev/mapper/1LIBVIRT3|/dev/mapper/1size%\\", \\"r%.*%\\" ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  back
up {  retain_min = 50  retain_days = 0 } " --deltag MDT_POOL_SPM_LVER=10 --deltag MDT__SHA_CKSUM=80a1d3d9853b52513e91f2c6566376c3e182e596 --addtag MDT__SHA_CKSUM=6024ad44f543655
d9654aa06c6ac7f394895234b --addtag MDT_POOL_SPM_LVER=11 c9f2fca8-35da-4d4c-ac73-9bd7748042e4' (cwd None)

....

Thread-22::ERROR::2011-04-27 10:21:17,351::task::855::TaskManager.Task::(_setError) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 863, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/storage/hsm.py", line 564, in public_connectStoragePool
    return pool.connect(hostID, scsiKey, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 398, in connect
    self.getMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1201, in getMasterDomain
    self.masterDomain = self.verifyMasterDomain(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1227, in verifyMasterDomain
    if not domain.isMaster():
  File "/usr/share/vdsm/storage/sd.py", line 658, in isMaster
    return self.getMetaParam(DMDK_ROLE).capitalize() == MASTER_DOMAIN
  File "/usr/share/vdsm/storage/sd.py", line 621, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 175, in __getitem__
    with self._accessWrapper():
  File "/usr/lib64/python2.6/contextlib.py", line 16, in __enter__
    return self.gen.next()
  File "/usr/share/vdsm/storage/persistentDict.py", line 125, in _accessWrapper
    self.refresh()
  File "/usr/share/vdsm/storage/persistentDict.py", line 245, in refresh
    raise se.MetaDataSealIsBroken(declaredChecksum, computedChecksum)
MetaDataSealIsBroken: Meta Data seal is broken (checksum mismatch): u'cksum = 6024ad44f543655d9654aa06c6ac7f394895234b, computed_cksum = 7791c76f93e5562395cac02d84961d94643b193c

Version-Release number of selected component (if applicable):
vdsm-4.9-62.el6.x86_64
lvm2-2.02.83-3.el6.x86_64

Additional info:
attached vdsm logs from both hosts.

Comment 4 Dan Kenigsberg 2011-05-16 10:58:46 UTC
http://gerrit.usersys.redhat.com/327

Comment 6 Kiril Nesenko 2011-06-14 09:40:49 UTC
Verified
ic122

Comment 7 errata-xmlrpc 2011-12-06 07:15:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html