Bug 1301681

Files in /var/lib/ovirt-hosted-engine-ha are autogenerated from the engine's OVF store. So yes, your changes will be overwritten.

This might be a duplicate of a feature bug we have, but I will have to find the number first...

Please reinstall the HE with correct settings to verify the other bug, this won't be fixed any time soon.

You can upload an image file with an updated borker.conf file. Undocumented but duable. Not very convenient as well.

Moran, this is kind of a regression in the behjavior introduced by the move to shred storage. We don't have an quick and easy way to change the content there. An API from the backend + a UI or something from the command line would be great.

(In reply to Roy Golan from comment #3)
> You can upload an image file with an updated borker.conf file. Undocumented
> but duable. Not very convenient as well.
> 
> Moran, this is kind of a regression in the behjavior introduced by the move
> to shred storage. We don't have an quick and easy way to change the content
> there. An API from the backend + a UI or something from the command line
> would be great.

Have you tried this? Did it worked for you?
I tried "vi" on the broker.conf at a time and then saving it, no success.
What's the path to the broker.conf at the shared storage?

Nikolai, there is no editable storage you could access with vi. The config file is compressed inside a ovf storage archive and extracted every time we want to read it. That is why we always overwrote your changes.

As Roy said, we need a UI or a specialized tool (or a hosted engine hack of some sort) to allow the modification.

(In reply to Martin Sivák from comment #5)
> Nikolai, there is no editable storage you could access with vi. The config
> file is compressed inside a ovf storage archive and extracted every time we
> want to read it. That is why we always overwrote your changes.
> 
> As Roy said, we need a UI or a specialized tool (or a hosted engine hack of
> some sort) to allow the modification.

Then creating some sort of hosted-engine --"vi broker.conf on shared storage" is expected.

I don't think that I can somehow possibly check it from here.

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Moving from 4.0 alpha to 4.0 beta since 4.0 alpha has been already released and bug is not ON_QA.

oVirt 4.0 beta has been released, moving to RC milestone.

oVirt 4.0 beta has been released, moving to RC milestone.

We created a regression in the infrastructure once we moved all the config to shared storage. We need to address that with Integ team

Do you have a RFE for this?

(In reply to Yaniv Dary from comment #12)
> Do you have a RFE for this?

No and we need an RFE for managing the HA cluster configuration.

Yaniv there is no way this would be solved to 4.0.

Bug tickets must have version flags set prior to targeting them to a release. Please ask maintainer to set the correct version flags and only then set the target milestone.

As a workaround I'm proposing a script to let the use upload a new file to the configurations volume.
A proper solution will require an utility or the capability to edit it from the engine.

Please ensure to take a backup of the configuration volume before using this:
 #!/bin/sh
 source /etc/ovirt-hosted-engine/hosted-engine.conf
 dir=`mktemp -d` && cd $dir
 systemctl stop ovirt-ha-broker # on all hosts!
 find /rhev/data-center/ -path "*/${sdUUID}/images/${conf_image_UUID}/${conf_volume_UUID}" -type f -exec sh -c 'sudo -u vdsm dd if=$1 2>/dev/null | tar -xvf - 2>/dev/null' {} {} \;
 # FIX your broker.conf as needed!!!
 find /rhev/data-center/ -path "*/${sdUUID}/images/${conf_image_UUID}/${conf_volume_UUID}" -type f -exec sh -c 'tar -cf- * | sudo -u vdsm dd of=$1 2>/dev/null' {} {} \;
 systemctl restart ovirt-ha-agent # on all hosts!

Btw I would also propose a small change to the notification mechanism here. Hosted engine should always use a localhost MTA to send emails. It then does not matter whether the MTA can reach network or not, hosted engine should be safe and sound and not affected by any timeouts or failures during the outbound smtp connection.

This means we need to somehow create the forwarding file based on the existing configuration (and document that /etc/aliases is the preferred way) and then hardcode localhost to the notification code.

The part that still needs to be configured somehow is the regex that controls which messages are going to be sent.

I think that we already discussed this option but, if I'm not wrong, we rejected because we didn't want to add an SMTP relay to each oVirt Node host.
Fabian?

The smtp relay system has to be asynchronous, period. High availability has unconditional priority here.

There are two ways to accomplish this:

1) Use well tested standard MTA
2) Write our own queuing thread

I strongly prefer number 1 as it allows use of all the standard mechanisms, especially /etc/aliases (sysadmins know how to use it, and other linux processes sometimes send emails to root user)

This bug had requires_doc_text flag, yet no documentation text was provided. Please add the documentation text and only then set this flag.

How should notifications configuration being added to environment?
Please supply reproduction for this bug.
In case of re-running engine-setup, I did not seen there any questions related to addition of notifications.
Tried this on latest rhevm-4.1.0-0.3.beta2.el7.noarch.

rhevm-spice-client-x64-msi-4.1-2.el7ev.noarch
rhevm-dependencies-4.1.0-0.1.alpha.gite3bec5a.el7ev.noarch
rhevm-spice-client-x86-msi-4.1-2.el7ev.noarch
rhevm-branding-rhev-4.1.0-0.0.master.20161128195345.el7ev.noarch
rhevm-4.1.0-0.3.beta2.el7.noarch
rhevm-doc-4.1.0-0.0.master.201611252145.el7ev.noarch
rhev-guest-tools-iso-4.1-3.el7ev.noarch
rhevm-guest-agent-common-1.0.12-3.el7ev.noarch
rhevm-setup-plugins-4.1.0-0.1.alpha.20161130141652.el7ev.noarch
Linux version 3.10.0-514.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:24:13 EDT 2016
Linux 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

(In reply to Nikolai Sednev from comment #21)
> How should notifications configuration being added to environment?
> Please supply reproduction for this bug.
> In case of re-running engine-setup, I did not seen there any questions
> related to addition of notifications.
> Tried this on latest rhevm-4.1.0-0.3.beta2.el7.noarch.

You can edit the notification settings using the new set-shared-config and get-shared-config actions as explained in detail here: 
http://www.ovirt.org/develop/release-management/features/sla/hosted-engine-edit-configuration-on-shared-storage/

(In reply to Jenny Tokar from comment #22)
> (In reply to Nikolai Sednev from comment #21)
> > How should notifications configuration being added to environment?
> > Please supply reproduction for this bug.
> > In case of re-running engine-setup, I did not seen there any questions
> > related to addition of notifications.
> > Tried this on latest rhevm-4.1.0-0.3.beta2.el7.noarch.
> 
> You can edit the notification settings using the new set-shared-config and
> get-shared-config actions as explained in detail here: 
> http://www.ovirt.org/develop/release-management/features/sla/hosted-engine-
> edit-configuration-on-shared-storage/

Nothing is working while following that documentstion:
"To get all keys and values:

    hosted-engine --get-all-config
    returns:
      type: broker  
      key: value
      key2: value"

On host I'm not getting any expected data except for original output of "hosted-engine --help" output.

# hosted-engine --get-shared-config key
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 51, in <module>
    value_and_type = get_shared_config.get_shared_config(*sys.argv)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 40, in get_shared_config
    result = ha_cli.get_shared_config(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 279, in get_shared_config
    return self._config.get_config_from_shared_storage(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 215, in get_config_from_shared_storage
    final_type = self._determine_final_config_type(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 241, in _determine_final_config_type
    "Configuration key not found: key={0}".format(key))
KeyError: 'Configuration key not found: key=key'

# hosted-engine --get-shared-config key --type broker
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 51, in <module>
    value_and_type = get_shared_config.get_shared_config(*sys.argv)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 40, in get_shared_config
    result = ha_cli.get_shared_config(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 279, in get_shared_config
    return self._config.get_config_from_shared_storage(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 215, in get_config_from_shared_storage
    final_type = self._determine_final_config_type(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 241, in _determine_final_config_type
    "Configuration key not found: key={0}".format(key))
KeyError: 'Configuration key not found: key=key'


Either http://www.ovirt.org/develop/release-management/features/sla/hosted-engine-edit-configuration-on-shared-storage/ is incomplete or something is broken within the code on these components on hosts:
rhvm-appliance-4.1.20170119.1-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.0-1.el7ev.noarch
ovirt-hosted-engine-setup-2.1.0-2.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-common-0.5.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64
mom-0.5.8-1.el7ev.noarch
vdsm-4.19.2-2.el7ev.x86_64
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-imageio-daemon-0.5.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Engine:
rhev-guest-tools-iso-4.1-3.el7ev.noarch
rhevm-doc-4.1.0-1.el7ev.noarch
rhevm-dependencies-4.1.0-1.el7ev.noarch
rhevm-setup-plugins-4.1.0-1.el7ev.noarch
rhevm-4.1.0.1-0.1.el7.noarch
rhevm-guest-agent-common-1.0.12-3.el7ev.noarch
rhevm-branding-rhev-4.1.0-0.el7ev.noarch
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

alma03 ~]# hosted-engine --get-shared-config key broker
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 51, in <module>
    value_and_type = get_shared_config.get_shared_config(*sys.argv)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 40, in get_shared_config
    result = ha_cli.get_shared_config(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 279, in get_shared_config
    return self._config.get_config_from_shared_storage(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 215, in get_config_from_shared_storage
    final_type = self._determine_final_config_type(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 241, in _determine_final_config_type
    "Configuration key not found: key={0}".format(key))
KeyError: 'Configuration key not found: key=key'
[root@alma03 ~]# hosted-engine --get-shared-config key --type broker
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 51, in <module>
    value_and_type = get_shared_config.get_shared_config(*sys.argv)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 40, in get_shared_config
    result = ha_cli.get_shared_config(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 279, in get_shared_config
    return self._config.get_config_from_shared_storage(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 215, in get_config_from_shared_storage
    final_type = self._determine_final_config_type(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 241, in _determine_final_config_type
    "Configuration key not found: key={0}".format(key))
KeyError: 'Configuration key not found: key=key'

" hosted-engine --get-all-config" not even exists:
alma03 ~]#  hosted-engine --get-all-config
Invalid option '--get-all-config'
Usage: /usr/sbin/hosted-engine [--help] <command> [<command-args>]
    --help
        show this help message.

    The available commands are:
        --deploy [options]
            run ovirt-hosted-engine deployment
        --vm-start
            start VM on this host
        --vm-start-paused
            start VM on this host with qemu paused
        --vm-shutdown
            gracefully shutdown the VM on this host
        --vm-poweroff
            forcefully poweroff the VM on this host
        --vm-status [--json]
            VM status according to the HA agent. If --json is given, the
            output will be in machine-readable (JSON) format
        --add-console-password [--password=<password>]
            Create a temporary password for vnc/spice connection.  If
            --password is given, the password will be set to the value
            provided.  Otherwise, if it is set,  the environment variable
            OVIRT_HOSTED_ENGINE_CONSOLE_PASSWORD will be used.  As a last
            resort, the password will be read interactively.
        --check-deployed
            Check whether the hosted engine has been deployed already
        --check-liveliness
            Checks liveliness page of engine
        --connect-storage
            Connect the storage domain
        --disconnect-storage
            Disconnect any connected storage domains
        --console
            Open the configured console using virsh on localhost
        --set-maintenance --mode=<mode>
            Set maintenance status to the specified mode (global/local/none)
        --set-shared-config key value [--type=<type>]
            Set specified key to the specified value. If the key is duplicated
            in several files a type must be provided.
        --get-shared-config key [--type=<type>]
            Get specified key's value. If the key is duplicated in several
            files a type must be provided.
        --reinitialize-lockspace
            Make sure all hosted engine agents are down and reinitialize the
            sanlock lockspaces.
        --clean-metadata
            Remove the metadata for the current host's agent from the global
            status database. This makes all other hosts forget about this
            host.
        --upgrade-appliance
            Upgrade the engine appliance.
        --rollback-upgrade
            Rollback using a backup saved in a previous upgrade attempt.

    For additional information about a specific command try:
         <command> --help

Sorry about the confusion, the --get-all-config should have appeared as not yet implemented. I'll change the feature page accordingly. 

The other options work, when you are running "hosted-engine --get-shared-config key --type broker" you are trying to get the value for a key "key" of the type "broker". Since there is no such key, you are getting an error.

(In reply to Jenny Tokar from comment #26)
> Sorry about the confusion, the --get-all-config should have appeared as not
> yet implemented. I'll change the feature page accordingly. 
> 
> The other options work, when you are running "hosted-engine
> --get-shared-config key --type broker" you are trying to get the value for a
> key "key" of the type "broker". Since there is no such key, you are getting
> an error.

What is the possible key value, which I should replace the "key" with and where are they documented?

From here: http://www.ovirt.org/develop/release-management/features/sla/hosted-engine-edit-configuration-on-shared-storage/

Editable keys for the first stage:

Broker.conf:

    smtp-server
    smtp-port
    source-email
    destination-emails
    state-transition

Before any SMTP configuration was available on environment:
hosted-engine --get-shared-config smtp-server --type=broker
broker : localhost, type : broker
hosted-engine --get-shared-config smtp-port --type=broker
broker : 25, type : broker
hosted-engine --get-shared-config source-email --type=broker
broker : root@localhost, type : broker
hosted-engine --get-shared-config destination-emails --type=broker
broker : root@localhost, type : broker

hosted-engine --get-shared-config state-transition --type=broker
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 51, in <module>
    value_and_type = get_shared_config.get_shared_config(*sys.argv)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/get_shared_config.py", line 40, in get_shared_config
    result = ha_cli.get_shared_config(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 279, in get_shared_config
    return self._config.get_config_from_shared_storage(key, config_type)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 216, in get_config_from_shared_storage
    value = self.get(final_type, key)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/env/config.py", line 210, in get
    key
KeyError: 'Configuration value not found: file=/var/lib/ovirt-hosted-engine-ha/broker.conf, key=state-transition'

Is state-transition is not working properly?

I also see that there is a difference in hosted-engine --help output for " key value" and "key" for "--set-shared-config" and "--get-shared-config".
Why it this inconsistency there?

--set-shared-config key value [--type=<type>]
            Set specified key to the specified value. If the key is duplicated
            in several files a type must be provided.
--get-shared-config key [--type=<type>]
            Get specified key's value. If the key is duplicated in several
            files a type must be provided.

This is not an inconsistency.
When setting the configuration you must provide a value, when getting the configuration you want to get the value. 
Or maybe I didn't understand the question?

Works for me on these components on host:
ovirt-hosted-engine-ha-2.1.0-1.el7ev.noarch
ovirt-host-deploy-1.6.0-1.el7ev.noarch
ovirt-imageio-common-0.5.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
qemu-kvm-rhev-2.6.0-28.el7_3.3.x86_64
libvirt-client-2.0.0-10.el7_3.4.x86_64
mom-0.5.8-1.el7ev.noarch
vdsm-4.19.2-2.el7ev.x86_64
ovirt-hosted-engine-setup-2.1.0-2.el7ev.noarch
ovirt-setup-lib-1.1.0-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-imageio-daemon-0.5.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
Linux version 3.10.0-514.6.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Sat Dec 10 11:15:38 EST 2016
Linux 3.10.0-514.6.1.el7.x86_64 #1 SMP Sat Dec 10 11:15:38 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)