Bug 842737

Summary: negative case - getSpmStatus keeps reporting "Meta Data parameter invalid" after connection to master domain was restored (stale cache)
Product: Red Hat Enterprise Virtualization Manager Reporter: Haim <hateya>
Component: vdsmAssignee: Ayal Baron <abaron>
Status: CLOSED WONTFIX QA Contact: Haim <hateya>
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: abaron, amureini, bazulay, hateya, iheim, lpeer, yeylon, ykaul
Target Milestone: ---   
Target Release: 3.1.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-04-04 15:14:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
vdsm.log none

Description Haim 2012-07-24 13:42:16 UTC
Description of problem:

I have a case of stale cache which prevents vdsm from re-connecting to pool, which looks as follows:

- local host with local storage domain (domain exists under /opt/dom1/<sdUUID>
- mv /opt/dom1/<sdUUID> /tmp/
- perform an SPM action 
- pool becomes non-responsive, engine send getSpmStatus to vdsm, vdsm returns:

Thread-117003::INFO::2012-07-24 19:27:11,860::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='13e32227-648f-4240-8eb1-45c1b4e423d4
', options=None)
Thread-117003::ERROR::2012-07-24 19:27:11,861::hsm::535::Storage.HSM::(getSpmStatus) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 527, in getSpmStatus
    status = {'spmStatus':pool.getSpmRole(), 'spmLver': pool.getSpmLver(), 'spmId':pool.getSpmId()}
  File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver
    return self.getMetaParam(PMDK_LVER)
  File "/usr/share/vdsm/storage/sp.py", line 1447, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 85, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 195, in __getitem__
    raise KeyError(key)
KeyError: 'POOL_SPM_LVER'
Thread-117003::ERROR::2012-07-24 19:27:11,861::task::853::TaskManager.Task::(_setError) Task=`e62289e7-e9b4-48bc-b135-e8f45ce5b8d7`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 861, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 536, in getSpmStatus
    raise se.MetaDataParamError("Version or spm id invalid")
MetaDataParamError: Meta Data parameter invalid: ('Version or spm id invalid',

- after that, I restore the storage domain to its original place
  * mv /tmp/sdUUID /opt/dom1/
- vdsm keeps reporting the above message forever although connection was restored, I even executed repoStats command and the results were positive:

 [root@nott-vds1 ~]# vdsClient -s 0 repoStats
Domain 77f493a8-078f-4e06-9506-4bd2ca4a8960 {'delay': '0.0029981136322', 'lastCheck': 1343141463.5053949, 'code': 0, 'valid': True}
Domain 7436be21-badb-4296-a274-c9dd010582f5 {'delay': '0.00148296356201', 'lastCheck': 1343141460.641876, 'code': 0, 'valid': True}
Domain 174d28bf-a3a6-49c8-9107-d3ffa2d5252c {'delay': '0.00143098831177', 'lastCheck': 1343141464.5150161, 'code': 0, 'valid': True}

[root@nott-vds1 ~]# less /var/log/vdsm/vdsm.log
[root@nott-vds1 ~]# vdsClient -s 0 getSpmStatus 13e32227-648f-4240-8eb1-45c1b4e423d4
Meta Data parameter invalid: ('Version or spm id invalid',)

The above test implies that vdsm reads from cache and not access to meta-data, in any-case, workaround  is to restart vdsm (which shows its a cache issue).

Comment 1 Haim 2012-07-24 13:44:13 UTC
Created attachment 600032 [details]
vdsm.log

Comment 5 RHEL Program Management 2012-12-14 07:59:01 UTC
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.