Description of problem:

I have a case of stale cache which prevents vdsm from re-connecting to pool, which looks as follows:

- local host with local storage domain (domain exists under /opt/dom1/<sdUUID>
- mv /opt/dom1/<sdUUID> /tmp/
- perform an SPM action 
- pool becomes non-responsive, engine send getSpmStatus to vdsm, vdsm returns:

Thread-117003::INFO::2012-07-24 19:27:11,860::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='13e32227-648f-4240-8eb1-45c1b4e423d4
', options=None)
Thread-117003::ERROR::2012-07-24 19:27:11,861::hsm::535::Storage.HSM::(getSpmStatus) Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/hsm.py", line 527, in getSpmStatus
    status = {'spmStatus':pool.getSpmRole(), 'spmLver': pool.getSpmLver(), 'spmId':pool.getSpmId()}
  File "/usr/share/vdsm/storage/sp.py", line 133, in getSpmLver
    return self.getMetaParam(PMDK_LVER)
  File "/usr/share/vdsm/storage/sp.py", line 1447, in getMetaParam
    return self._metadata[key]
  File "/usr/share/vdsm/storage/persistentDict.py", line 85, in __getitem__
    return dec(self._dict[key])
  File "/usr/share/vdsm/storage/persistentDict.py", line 195, in __getitem__
    raise KeyError(key)
KeyError: 'POOL_SPM_LVER'
Thread-117003::ERROR::2012-07-24 19:27:11,861::task::853::TaskManager.Task::(_setError) Task=`e62289e7-e9b4-48bc-b135-e8f45ce5b8d7`::Unexpected error
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 861, in _run
    return fn(*args, **kargs)
  File "/usr/share/vdsm/logUtils.py", line 38, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 536, in getSpmStatus
    raise se.MetaDataParamError("Version or spm id invalid")
MetaDataParamError: Meta Data parameter invalid: ('Version or spm id invalid',

- after that, I restore the storage domain to its original place
  * mv /tmp/sdUUID /opt/dom1/
- vdsm keeps reporting the above message forever although connection was restored, I even executed repoStats command and the results were positive:

 [root@nott-vds1 ~]# vdsClient -s 0 repoStats
Domain 77f493a8-078f-4e06-9506-4bd2ca4a8960 {'delay': '0.0029981136322', 'lastCheck': 1343141463.5053949, 'code': 0, 'valid': True}
Domain 7436be21-badb-4296-a274-c9dd010582f5 {'delay': '0.00148296356201', 'lastCheck': 1343141460.641876, 'code': 0, 'valid': True}
Domain 174d28bf-a3a6-49c8-9107-d3ffa2d5252c {'delay': '0.00143098831177', 'lastCheck': 1343141464.5150161, 'code': 0, 'valid': True}

[root@nott-vds1 ~]# less /var/log/vdsm/vdsm.log
[root@nott-vds1 ~]# vdsClient -s 0 getSpmStatus 13e32227-648f-4240-8eb1-45c1b4e423d4
Meta Data parameter invalid: ('Version or spm id invalid',)

The above test implies that vdsm reads from cache and not access to meta-data, in any-case, workaround  is to restart vdsm (which shows its a cache issue).