Created attachment 1421281 [details]
ovirt-engine logs

Description of problem:
During automation tests was found multiple repeated exceptions in vdsm.log:

vdsm[57730]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo  
    stats = instance.get_all_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats
    with broker.connection(self._retries, self._wait):
  File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 99, in connection
    self.connect(retries, wait)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 78, in connect
    raise BrokerConnectionError(error_msg)
  BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)

vdsm[62989]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo
    stats = instance.get_all_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 106, in get_all_stats
    stats = broker.get_stats_from_storage(service)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 233, in get_stats_from_storage
    result = self._checked_communicate(request)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 261, in _checked_communicate
    .format(message or response))
RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/blockSD/4fcb4891-17ad-4832-a088-36f5a2f34cee/ha_agent/hosted-engine.metadata'

Issue was found on hosts: lynx22 and lynx23

Version-Release number of selected component (if applicable):
rhv-4.1.11-1
vdsm-4.19.50-1.el7ev.x86_64

How reproducible:
Found during automation

Steps to Reproduce:
Unclear - it happens during some different tests

Actual results:
Multiple repeated exceptions in the logs

Expected results:
None exceptions in the logs

Additional info:
attached logs from tests