Bug 1352286

Summary: Addition of additional hosted-engine-host failed via REST.
Product: [oVirt] ovirt-host-deploy Reporter: Nikolai Sednev <nsednev>
Component: CoreAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED DUPLICATE QA Contact: Pavel Stehlik <pstehlik>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: masterCC: bugs, dfediuck, nsednev, oourfali, rgolan, stirabos, ylavi
Target Milestone: ---Keywords: Regression
Target Release: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-06 10:11:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1200469    
Attachments:
Description Flags
sosreport from host alma03
none
sosreport from engine none

Description Nikolai Sednev 2016-07-03 11:26:28 UTC 
Description of problem:
Addition of additional hosted-engine-host failed.
I've tried to add additional hosted-engine-host using REST-API and addition had failed.
 

Version-Release number of selected component (if applicable):
Host:
ovirt-vmconsole-host-1.0.3-1.el7ev.noarch
ovirt-hosted-engine-ha-2.0.0-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.7.0-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.5.x86_64
ovirt-host-deploy-1.5.0-1.el7ev.noarch
ovirt-hosted-engine-setup-2.0.0.2-1.el7ev.noarch
ovirt-setup-lib-1.0.2-1.el7ev.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.17.x86_64
mom-0.5.5-1.el7ev.noarch
ovirt-vmconsole-1.0.3-1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
vdsm-4.18.5.1-1.el7ev.x86_64
rhev-release-4.0.1-1-001.noarch
sanlock-3.2.4-2.el7_2.x86_64
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch
Linux version 3.10.0-327.28.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon Jun 27 14:48:28 EDT 2016
Linux 3.10.0-327.28.2.el7.x86_64 #1 SMP Mon Jun 27 14:48:28 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

Engine:
rhevm-doc-4.0.0-2.el7ev.noarch
rhevm-setup-plugins-4.0.0.1-1.el7ev.noarch
rhevm-spice-client-x64-msi-4.0-2.el7ev.noarch
rhevm-4.0.2-0.2.rc1.el7ev.noarch
rhev-release-4.0.0-19-001.noarch
rhev-release-4.0.1-1-001.noarch
rhevm-guest-agent-common-1.0.12-2.el7ev.noarch
rhevm-dependencies-4.0.0-1.el7ev.noarch
rhevm-branding-rhev-4.0.0-2.el7ev.noarch
rhevm-spice-client-x86-msi-4.0-2.el7ev.noarch
rhev-guest-tools-iso-4.0-2.el7ev.noarch
Linux version 3.10.0-327.22.2.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Thu Jun 9 10:09:10 EDT 2016
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)



How reproducible:
100%

Steps to Reproduce:
1.Deploy new HE environment on one host over NFS.
2.Add additional hosted-engine-host using REST-API.
3.

Actual results:
Addition of new hosted engine host failed.

Expected results:
Additional hosted-engine-host should be successfully added to hosted-engine.

Additional info:
Comment 1 Nikolai Sednev 2016-07-03 11:29:27 UTC 
Created attachment 1175516 [details]
sosreport from host alma03
Comment 2 Nikolai Sednev 2016-07-03 11:30:36 UTC 
Created attachment 1175517 [details]
sosreport from engine
Comment 3 Oved Ourfali 2016-07-03 11:36:05 UTC 
I see this error repeats on the host.
Moving to integration to analyze this one.
Sandro - Contact us if you need any assistance, or if you suspect an infra issue.

periodic/1::ERROR::2016-07-03 14:12:02,186::api::253::root::(_getHaInfo) failed to retrieve Hosted Engine HA info
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 232, in _getHaInfo
    stats = instance.get_all_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats
    self._configure_broker_conn(broker)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn
    dom_type=dom_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 176, in set_storage_domain
    .format(sd_type, options, e))
RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'nfs3', 'sd_uuid': '29d459ea-989d-4127-b996-248928adf543'}: Request failed: <class 'ovirt_hosted_engine_ha.lib.storage_backends.BackendFailureException'>
Comment 4 Yaniv Kaul 2016-07-04 06:51:11 UTC 
Did the same scenario work via the UI?
Comment 5 Simone Tiraboschi 2016-07-04 07:06:24 UTC 
(In reply to Yaniv Kaul from comment #4)
> Did the same scenario work via the UI?

I fear so.
On my opinion this is just a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1349532 (NGN 4.0, UI).
It's probably a race condition so it could be not that systematic but, if I correctly understood it ( https://bugzilla.redhat.com/show_bug.cgi?id=1349532#c14 ), we can hit on NGN and el hosts and from REST or from the UI.
Comment 6 Roy Golan 2016-07-06 10:11:27 UTC 

*** This bug has been marked as a duplicate of bug 1349532 ***