Bug 1439069 - If user tries to reinstall any node once the primary gluster node is replaced contents of /etc/ovirt-hosted-engine/hosted-engine.conf are overwritten
Summary: If user tries to reinstall any node once the primary gluster node is replaced...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.Gluster
Version: 2.1.0.4
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ovirt-4.1.8
: ---
Assignee: Sahina Bose
QA Contact: RamaKasturi
URL:
Whiteboard:
Depends On: 1457357
Blocks: Gluster-HC-3 1433896 1489345
TreeView+ depends on / blocked
 
Reported: 2017-04-05 07:23 UTC by RamaKasturi
Modified: 2019-04-28 13:13 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1489345 (view as bug list)
Environment:
Last Closed: 2017-12-11 16:29:26 UTC
oVirt Team: Gluster
Embargoed:
sabose: ovirt-4.1?
sabose: planning_ack?
rule-engine: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)

Description RamaKasturi 2017-04-05 07:23:51 UTC
Description of problem:
Once user replaces the primary gluster node and if he tries to reinstall any nodes in the cluster /etc/ovirt-hosted-engine/hosted-engine.conf contents are overwritten. I see that value for "storage" is overwritten and host fails to mount the storage and goes to nonoperational.

Version-Release number of selected component (if applicable):
Red Hat Virtualization Manager Version: 4.1.1.2-0.1.el7

How reproducible:
Always

Steps to Reproduce:
1. Replace primary gluster server in HC cluster
2. Now try reinstalling any node in the cluster
3.

Actual results:
I see that /etc/ovirt-hosted-engine/hosted-engine.conf file gets overridden. value of storage points to the old host which is removed and due to this host is not able to mount storage and goes to non operational state.

Expected results:
/etc/ovirt-hosted-engine/hosted-engine.conf file should not be overwritten when a host is re-installed.

Comment 1 Sahina Bose 2017-04-05 13:13:20 UTC
Is there a workaround?

Comment 2 RamaKasturi 2017-04-06 10:47:48 UTC
sahina, i moved the node to maintenance changed the contents of /etc/ovirt-hosted-engine.conf file to point to correct storage path, rebooted the node and activated it which brought the node online and mounted all the volumes back.

Comment 3 RamaKasturi 2017-04-13 13:45:34 UTC
If user retries Hosted Engine deployment for any reason, /etc/ovirt-hosted-engine/hosted-engine.conf file will not contain the mnt_options. user can later on edit this file to set the mnt_options. But if user tries to reinstall any of the node from UI, mnt_options from the conf file will be lost.

Comment 4 Sahina Bose 2017-05-30 07:27:15 UTC
Simone, are we adding UI enhancements to edit the conf file? Should this bug be retargeted?

Comment 5 Simone Tiraboschi 2017-05-31 14:54:06 UTC
The engine fetches that value from the initial answerfile stored on a configuration volume on the hosted-engine storage domain. We have an helper utility ('hosted-engine --set-shared-config') to let the user edit values there but currently he can just modify notifications related values.

Comment 6 Yaniv Kaul 2017-08-04 16:06:52 UTC
Is this on track to get into 4.1.5?

Comment 7 Simone Tiraboschi 2017-08-07 09:37:58 UTC
(In reply to Yaniv Kaul from comment #6)
> Is this on track to get into 4.1.5?

No code changes are required here: once 1457357 got it (and we are going to have it in 4.1.5) it's just a matter of documenting how to use it.

Comment 8 Sahina Bose 2017-10-11 06:46:34 UTC
Moving this to ON_QA as bug 1457357 is already in.

Comment 10 RamaKasturi 2017-11-06 07:25:43 UTC
Thanks sahina!!! We will get this verified for 4.1.8

Comment 11 RamaKasturi 2017-12-04 10:10:31 UTC
Moving this back to assigned state as i see that once the primary node is replaced and if user tries to reinstall any other node i see that /etc/ovirt-hosted-engine/hosted-engine.conf gets overwritten to the older value. This time i see that host does not move to non operational instead i see that HA score does not appear and running hosted-engine --vm-status gives the error below.

[root@zod ~]# hosted-engine --vm-status
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 180, in <module>
    if not status_checker.print_status():
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 104, in print_status
    all_host_stats = self._get_all_host_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 73, in _get_all_host_stats
    all_host_stats = ha_cli.get_all_host_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats
    return self.get_all_stats(self.StatModes.HOST)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats
    stats = broker.get_stats_from_storage(service)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 233, in get_stats_from_storage
    result = self._checked_communicate(request)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 261, in _checked_communicate
    .format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/10.70.36.78:_engine/7f8bb254-e645-4152-898a-fe711af9fd53/ha_agent/hosted-engine.metadata'

Comment 12 Red Hat Bugzilla Rules Engine 2017-12-04 10:10:36 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 13 Sahina Bose 2017-12-04 11:24:16 UTC
Moving this back to QA as editing storage domain option should work

Comment 14 RamaKasturi 2017-12-04 12:04:21 UTC
Verified and works fine with build ovirt-hosted-engine-setup-2.1.4-1.el7ev.noarch

During the replacement of primary node to set the storage path, instead of editing the  file /etc/ovirt-hosted-engine/hosted-engine.conf manually there is an option to set it via hosted-engine --set-shared-config and the value can be set as below[1] .

When the value is set as shown below it does not get rewritten during reinstall of the host.

[1] hosted-engine --set-shared-config storage <ip_of_the_replaced_node>:/engine>


Note You need to log in before you can comment on or make changes to this bug.