Bug 1323969

Description of problem:
Could be split between virt and infra, targeting infra for the time being
Experienced only during recovery only while tinkering with containers (https://github.com/mojaves/ovirt-site/blob/container-support/source/feature/container-support.html.md)
I believe this is triggered by recovery being much faster with containers. Not sure it is a real risk using VMs.

We need to wait for JSONRPC binding availability before to start recovery, otherwise VM recovery will fail at the final step because

Thread-12::DEBUG::2016-04-05 10:11:22,733::__init__::207::jsonrpc.Notification::(emit) Sending event {"params": {"notify_time": 4376331650, "dd38b452-0a9a-4e9e-a349-2f4ca73cc8c5": {"status": "Up", "displayInfo": [{"tlsPort": "-1", "ipAddress": "0", "type": "vnc", "port": "-1"}], "hash": "-8241192262902672289", "cpuUser": "0.00", "displayIp": "0", "monitorResponse": "0", "elapsedTime": "630", "displayType": "vnc", "cpuSys": "0.00", "displayPort": "-1", "displaySecurePort": "-1", "timeOffset": "0", "clientIp": ""}}, "jsonrpc": "2.0", "method": "|virt|VM_status|dd38b452-0a9a-4e9e-a349-2f4ca73cc8c5"}
Thread-12::ERROR::2016-04-05 10:11:22,733::vm::759::virt.vm::(_startUnderlyingVm) vmId=`dd38b452-0a9a-4e9e-a349-2f4ca73cc8c5`::The vm start process failed
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 740, in _startUnderlyingVm
    self.send_status_event(**self._getRunningVmStats())
  File "/usr/share/vdsm/virt/vm.py", line 349, in send_status_event
    self._notify('VM_status', stats)
  File "/usr/share/vdsm/virt/vm.py", line 353, in _notify
    self.cif.notify(sub_id, **{self.id: params})
  File "/usr/share/vdsm/clientIF.py", line 160, in notify
    notification.emit(**kwargs)
  File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 208, in emit
    self._cb(notification)
  File "/usr/share/vdsm/clientIF.py", line 163, in _send_notification
    self.bindings['jsonrpc'].reactor.server.send(message,
KeyError: 'jsonrpc'

The binding will be booted only later:

MainThread::DEBUG::2016-04-05 10:11:22,835::protocoldetector::206::vds.MultiProtocolAcceptor::(add_detector) Adding detector <yajsonrpc.stompreactor.StompDetector instance at 0x1696830>
MainThread::DEBUG::2016-04-05 10:11:22,836::executor::69::Executor::(start) Starting executor
MainThread::DEBUG::2016-04-05 10:11:22,837::executor::161::Executor::(__init__) Starting worker jsonrpc.Executor/0
MainThread::DEBUG::2016-04-05 10:11:22,837::executor::161::Executor::(__init__) Starting worker jsonrpc.Executor/1


Version-Release number of selected component (if applicable):
Vdsm master

How reproducible:
I've only seen that using containers.

Steps to Reproduce:
Hopefully there are shorter ways, however that's how I do it:
1. Patch Vdsm to run containers: https://github.com/mojaves/ovirt-site/blob/container-support/source/feature/container-support.html.md#how-to-try-the-feature
2. run 1+ container
3. restart Vdsm

Actual results:
Vdsm fails to recovery with the aforementioned stack trace

Expected results:
Vdsm recovers succesfully

Additional info:
Please check full Vdsm log attached