Bug 903673

There is a race in (vdsm/supervdsm.py):

    def isRunning(self):
        try:
            with open(self.pidfile, "r") as f:
                spid = f.read().strip()
            with open(self.timestamp, "r") as f:
                createdTime = f.read().strip()
        except IOError as e:
            # pid file and timestamp file must be exist after first launch,
            # otherwise excpetion will be raised to svdsm caller
            if e.errno == ENOENT and self._firstLaunch:
                return False
            else:
                raise

        try:
            pTime = str(misc.getProcCtime(spid))
        except OSError as e:
            if e.errno == ESRCH:
                # Means pid is not exist, svdsm was killed
                return False
            else:
                raise

        if pTime == createdTime:
            return True
        else:
            return False

When vdsm starts the isRunning check would be successful also for a previous instance of superVdsm (to which vdsm is not connected).
In ProxyCaller then we would skip the launch part going straight to "return callMethod()":

class ProxyCaller(object):
    def __call__(self, *args, **kwargs):
        callMethod = lambda: \
            getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
                                                                 **kwargs)
        if not self._supervdsmProxy.isRunning():
            # getting inside only when svdsm is down. its rare case so we
            # don't care that isRunning will run twice
            with self._supervdsmProxy.proxyLock:
                if not self._supervdsmProxy.isRunning():
                    self._supervdsmProxy.launch()

        try:
            return callMethod()

This has been also verified adding the following debug line to isRunning:

 self._log.debug("BZ903673: %s %s %s", spid, pTime, createdTime)

And the logs reported that indeed vdsm at the startup found a previous superVdsm instance still running:

PolicyEngine::DEBUG::2013-01-24 17:53:45,170::supervdsm::166::SuperVdsmProxy::(isRunning) BZ903673: 14545 1359042231.31 1359042231.31
[...]
MainThread::INFO::2013-01-24 17:54:15,788::vdsm::88::vds::(run) I am the actual vdsm 4.10-1.2 orange-vdsf.qa.lab.tlv.redhat.com (2.6.32-355.el6.x86_64)
[...]
MainThread::DEBUG::2013-01-24 17:54:16,125::supervdsm::166::SuperVdsmProxy::(isRunning) BZ903673: 14545 1359042231.31 1359042231.31
MainThread::ERROR::2013-01-24 17:54:16,125::clientIF::260::vds::(_initIRS) Error initializing IRS
Traceback (most recent call last):
  File "/usr/share/vdsm/clientIF.py", line 258, in _initIRS
    self.irs = Dispatcher(HSM())
  File "/usr/share/vdsm/storage/hsm.py", line 349, in __init__
    if not multipath.isEnabled():
  File "/usr/share/vdsm/storage/multipath.py", line 90, in isEnabled
    mpathconf = svdsm.readMultipathConf()
  File "/usr/share/vdsm/supervdsm.py", line 76, in __call__
    return callMethod()
  File "/usr/share/vdsm/supervdsm.py", line 66, in <lambda>
    getattr(self._supervdsmProxy._svdsm, self._funcName)(*args,
AttributeError: 'ProxyCaller' object has no attribute 'readMultipathConf'

Probably a duplicate of bug 890365

yes. same issue, after blocking storage connectivity, vdsm is restarted and the bug caused a race by using internal uninitialized variable.

This is the fix for that issue - http://gerrit.ovirt.org/#/c/10491

*** This bug has been marked as a duplicate of bug 890365 ***