Bug 1566984

Summary: vdsm root ERROR failed to retrieve Hosted Engine HA info exceptions in logs
Product: [oVirt] vdsm Reporter: Petr Kubica <pkubica>
Component: GeneralAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED WONTFIX QA Contact: meital avital <mavital>
Severity: medium Docs Contact:
Priority: high    
Version: 4.19.41CC: bugs, dfediuck, lsvaty, ratamir, ylavi
Target Milestone: ---Keywords: Automation, AutomationBlocker, Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-17 16:02:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
ovirt-engine logs none

Description Petr Kubica 2018-04-13 09:17:11 UTC
Created attachment 1421281 [details]
ovirt-engine logs

Description of problem:
During automation tests was found multiple repeated exceptions in vdsm.log:

vdsm[57730]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 231, in _getHaInfo  
    stats = instance.get_all_stats()
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py&quot;, line 103, in get_all_stats
    with broker.connection(self._retries, self._wait):
  File &quot;/usr/lib64/python2.7/contextlib.py&quot;, line 17, in __enter__
    return self.gen.next()
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 99, in connection
    self.connect(retries, wait)
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 78, in connect
    raise BrokerConnectionError(error_msg)
  BrokerConnectionError: Failed to connect to broker, the number of errors has exceeded the limit (1)

vdsm[62989]: vdsm root ERROR failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
  File &quot;/usr/lib/python2.7/site-packages/vdsm/host/api.py&quot;, line 231, in _getHaInfo
    stats = instance.get_all_stats()
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py&quot;, line 106, in get_all_stats
    stats = broker.get_stats_from_storage(service)
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 233, in get_stats_from_storage
    result = self._checked_communicate(request)
  File &quot;/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py&quot;, line 261, in _checked_communicate
    .format(message or response))
RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: &apos;/rhev/data-center/mnt/blockSD/4fcb4891-17ad-4832-a088-36f5a2f34cee/ha_agent/hosted-engine.metadata&apos;

Issue was found on hosts: lynx22 and lynx23

Version-Release number of selected component (if applicable):
rhv-4.1.11-1
vdsm-4.19.50-1.el7ev.x86_64

How reproducible:
Found during automation

Steps to Reproduce:
Unclear - it happens during some different tests

Actual results:
Multiple repeated exceptions in the logs

Expected results:
None exceptions in the logs

Additional info:
attached logs from tests

Comment 4 Red Hat Bugzilla Rules Engine 2018-04-16 07:32:44 UTC
This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

Comment 5 Simone Tiraboschi 2018-04-16 15:08:59 UTC
Unable to reproduce with node 0 flow, not sure we want to fix for the vintage flow.

Comment 6 Simone Tiraboschi 2018-04-17 16:02:38 UTC
Not reproducible with node zero and the vintage flow is deprecated.