Bug 1422962 - [downstream clone - 3.6.11] [Hosted-Engine] 3.5 HE SD upgrade fails if done on initial host
Summary: [downstream clone - 3.6.11] [Hosted-Engine] 3.5 HE SD upgrade fails if done o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-ha
Version: 3.6.9
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-3.6.11
: ---
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On: 1422864
Blocks: 1400800 1434954
TreeView+ depends on / blocked
 
Reported: 2017-02-16 16:37 UTC by rhev-integ
Modified: 2020-04-15 15:18 UTC (History)
9 users (show)

Fixed In Version: ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch
Doc Type: Bug Fix
Doc Text:
With this update, incorrect permissions causing the Red Hat Enterprise Virtualization 3.5 self-hosted engine storage domain upgrade to fail if it was performed on the initial host have been corrected.
Clone Of: 1422864
Environment:
Last Closed: 2017-05-09 17:05:03 UTC
oVirt Team: Integration
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
engine (7.65 MB, application/x-xz)
2017-04-12 20:39 UTC, Nikolai Sednev 		no flags Details
sosreport-puma18 (14.45 MB, application/x-xz)
2017-04-12 20:41 UTC, Nikolai Sednev 		no flags Details
sosreport-puma19 (8.74 MB, application/x-xz)
2017-04-12 20:42 UTC, Nikolai Sednev 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2792381 0 None None None 2017-02-16 16:38:08 UTC
Red Hat Product Errata RHBA-2017:1213 0 normal SHIPPED_LIVE ovirt-hosted-engine-ha bug fix update for 3.6.11 2017-05-09 21:03:43 UTC
oVirt gerrit 72409 0 None None None 2017-02-16 16:38:08 UTC
oVirt gerrit 72434 0 None None None 2017-02-16 16:38:08 UTC
oVirt gerrit 1422962 0 None None None 2017-02-20 08:47:50 UTC

Description rhev-integ 2017-02-16 16:37:45 UTC 
+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1422864 +++
======================================================================

+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1401359 +++
======================================================================

Description of problem:

In 3.5, /etc/ovirt-hosted-engine/answers.conf permissions are, right after install, as below:

Initial Host:
-rw-rw----. 1 root root 2585 Dec  5 00:08 /etc/ovirt-hosted-engine/answers.conf

Additional Hosts:
-rw-r--r--. 1 root root 2575 Dec  5 00:18 /etc/ovirt-hosted-engine/answers.conf

When upgrading to 3.6, if the Host chosen to upgrade first (to ha 1.3.x) is the Initial one selected for the initial deployment of HE (-rw-rw----), the HE SD upgrade fails due to EACCESS to answers.conf file.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160920.1.el7ev)
ovirt-hosted-engine-ha-1.3.5.8-1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy fresh Hosted Engine on 3.5 using 20160219.0.el7ev
2. Upgrade initial HE host to 20160920.1.el7ev
3. Trigger HE SD Upgrade (Host in maintenance, restart ha-agent)

Actual results:
If one chooses the initial HE Host to do the upgrade, the HE SD not Upgraded, ha-agent keeps restarting. See:
MainThread::INFO::2016-12-05 01:24:38,917::upgrade::1010::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Upgrading to current version
MainThread::INFO::2016-12-05 01:24:39,004::upgrade::736::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_stopMonitoringDomain) Stop monitoring domain
MainThread::INFO::2016-12-05 01:24:39,059::upgrade::151::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Looking for conf volume
MainThread::ERROR::2016-12-05 01:24:39,112::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume
MainThread::INFO::2016-12-05 01:24:39,112::upgrade::953::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) _move_to_shared_conf
MainThread::INFO::2016-12-05 01:24:39,112::upgrade::375::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Reading conf file: fhanswers.conf
MainThread::ERROR::2016-12-05 01:24:39,112::upgrade::399::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2016-12-05 01:24:39,113::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent

Expected results:
HE SD upgraded. 

Additional info:
The permission issue seem to be there for some time. In previous hosted-engine-ha in 3.6 apparently used to upgrade the HE SD even when hitting this "Permission denied" error. See: https://bugzilla.redhat.com/show_bug.cgi?id=1292652#c1

Apparently we missed that error in that BZ, and now the behavior is slightly different, we restart the agent and the HE SD is NOT upgraded at the step it should. This causes troubles for 3.5 to 3.6 Upgrade.

(Originally by Germano Veit Michel)

(Originally by rhev-integ)
Comment 6 Nikolai Sednev 2017-04-12 20:38:43 UTC 
1)Deployed clean environment on puma18 and puma19 hosts, over NFS storage domain and added two NFS data storage domains.
Components on hosts:
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
sanlock-3.2.4-3.el7_2.x86_64
rhevm-sdk-python-3.5.6.0-1.el7ev.noarch
mom-0.4.1-4.el7ev.noarch
vdsm-4.16.38-1.el7ev.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.25.x86_64
ovirt-hosted-engine-ha-1.2.10-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.6.x86_64
ovirt-host-deploy-1.3.2-1.el7ev.noarch
Linux version 3.10.0-327.53.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 10:49:09 EDT 2017
Linux 3.10.0-327.53.1.el7.x86_64 #1 SMP Tue Mar 14 10:49:09 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

On engine:
rhevm-lib-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.5.8-0.1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch
rhevm-setup-plugins-3.5.4-1.el6ev.noarch
rhevm-iso-uploader-3.5.1-1.el6ev.noarch
rhevm-extensions-api-impl-3.5.8-0.1.el6ev.noarch
rhevm-spice-client-x64-msi-3.5-3.el6.noarch
rhevm-backend-3.5.8-0.1.el6ev.noarch
rhevm-sdk-python-3.5.6.0-1.el6ev.noarch
ovirt-host-deploy-1.3.2-1.el6ev.noarch
rhevm-spice-client-x64-cab-3.5-3.el6.noarch
rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch
rhevm-setup-base-3.5.8-0.1.el6ev.noarch
rhevm-reports-3.5.8-1.el6ev.noarch
rhevm-image-uploader-3.5.0-4.el6ev.noarch
rhevm-setup-plugin-websocket-proxy-3.5.8-0.1.el6ev.noarch
ovirt-host-deploy-java-1.3.2-1.el6ev.noarch
rhevm-spice-client-x86-cab-3.5-3.el6.noarch
rhevm-setup-3.5.8-0.1.el6ev.noarch
rhevm-tools-3.5.8-0.1.el6ev.noarch
rhevm-3.5.8-0.1.el6ev.noarch
rhevm-log-collector-3.5.4-2.el6ev.noarch
rhev-guest-tools-iso-3.5-15.el6ev.noarch
rhevm-doc-3.5.3-1.el6eng.noarch
rhevm-spice-client-x86-msi-3.5-3.el6.noarch
rhevm-guest-agent-common-1.0.10-2.el6ev.noarch
rhevm-dwh-setup-3.5.5-1.el6ev.noarch
rhevm-branding-rhev-3.5.0-4.el6ev.noarch
rhevm-cli-3.5.0.6-1.el6ev.noarch
rhevm-dbscripts-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.5.8-0.1.el6ev.noarch
rhevm-reports-setup-3.5.8-1.el6ev.noarch
rhevm-dependencies-3.5.1-1.el6ev.noarch
rhevm-websocket-proxy-3.5.8-0.1.el6ev.noarch
rhevm-userportal-3.5.8-0.1.el6ev.noarch
rhevm-restapi-3.5.8-0.1.el6ev.noarch
Linux version 2.6.32-573.41.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Mar 2 11:08:17 EST 2017
Linux 2.6.32-573.41.1.el6.x86_64 #1 SMP Thu Mar 2 11:08:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)

puma18 ~]# hosted-engine --vm-status


--== Host 1 status ==--

Status up-to-date                  : True
Hostname                           : puma18
Host ID                            : 1
Engine status                      : {"health": "good", "vm": "up", "detail": "up"}
Score                              : 2400
Local maintenance                  : False
Host timestamp                     : 21579
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=21579 (Wed Apr 12 19:11:01 2017)
        host-id=1
        score=2400
        maintenance=False
        state=EngineUp


--== Host 2 status ==--

Status up-to-date                  : True
Hostname                           : puma19
Host ID                            : 2
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 2400
Local maintenance                  : False
Host timestamp                     : 21438
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=21438 (Wed Apr 12 19:11:02 2017)
        host-id=2
        score=2400
        maintenance=False
        state=EngineDown


2)Permissions right after deployment on both hosts as appears bellow:
puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K Apr 12 17:52 /etc/ovirt-hosted-engine/answers.conf
puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root root 2.6K Apr 12 18:01 /etc/ovirt-hosted-engine/answers.conf

3)Created one guest VM with RHEL7.3 OS.
4)Upgraded HE from 3.5 to 3.6 following https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html
5)I was able to reproduce this issue on my environment.
MainThread::INFO::2017-04-12 22:24:36,343::upgrade::1012::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Upgrading to current version
MainThread::INFO::2017-04-12 22:24:36,368::upgrade::738::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_stopMonitoringDomain) Stop monitoring domain
MainThread::INFO::2017-04-12 22:24:36,383::upgrade::153::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Looking for conf volume
MainThread::ERROR::2017-04-12 22:24:36,399::upgrade::209::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume
MainThread::INFO::2017-04-12 22:24:36,399::upgrade::955::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) _move_to_shared_conf
MainThread::INFO::2017-04-12 22:24:36,400::upgrade::377::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Reading conf file: fhanswers.conf
MainThread::ERROR::2017-04-12 22:24:36,400::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-12 22:24:36,400::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent

Moving back to assigned.

I was not able to move the second host to maintenance as HE-VM was running on it and it was failing to migrate to 3.6's rhel7.3 puma18.

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K Apr 12 17:52 /etc/ovirt-hosted-engine/answers.conf
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017
Linux puma18 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)
Comment 7 Nikolai Sednev 2017-04-12 20:39:44 UTC 
Created attachment 1271264 [details]
engine
Comment 8 Nikolai Sednev 2017-04-12 20:41:06 UTC 
Created attachment 1271265 [details]
sosreport-puma18
Comment 9 Nikolai Sednev 2017-04-12 20:42:17 UTC 
Created attachment 1271266 [details]
sosreport-puma19
Comment 10 Simone Tiraboschi 2017-04-13 09:53:38 UTC 
ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch didn't included the fix, I built ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch with it
Comment 11 Nikolai Sednev 2017-04-25 15:48:35 UTC 
I still don't see ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch being distributed to latest 3.6.

Still being reproduced on latest 3.6 components on host:
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)


'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent
MainThread::ERROR::2017-04-25 18:41:40,260::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-25 18:41:40,260::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent
MainThread::ERROR::2017-04-25 18:41:46,008::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-25 18:41:46,008::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent
Comment 18 Nikolai Sednev 2017-05-03 14:47:05 UTC 
1)
Deployed clean 3.5.8-0.1.el6ev hosted engine over NFS:
rhevm-lib-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.5.8-0.1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch
rhevm-setup-plugins-3.5.4-1.el6ev.noarch
rhevm-iso-uploader-3.5.1-1.el6ev.noarch
rhevm-extensions-api-impl-3.5.8-0.1.el6ev.noarch
rhevm-spice-client-x64-msi-3.5-3.el6.noarch
rhevm-backend-3.5.8-0.1.el6ev.noarch
rhevm-sdk-python-3.5.6.0-1.el6ev.noarch
ovirt-host-deploy-1.3.2-1.el6ev.noarch
rhevm-spice-client-x64-cab-3.5-3.el6.noarch
rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch
rhevm-setup-base-3.5.8-0.1.el6ev.noarch
rhevm-reports-3.5.8-1.el6ev.noarch
rhevm-image-uploader-3.5.0-4.el6ev.noarch
rhevm-setup-plugin-websocket-proxy-3.5.8-0.1.el6ev.noarch
ovirt-host-deploy-java-1.3.2-1.el6ev.noarch
rhevm-spice-client-x86-cab-3.5-3.el6.noarch
rhevm-setup-3.5.8-0.1.el6ev.noarch
rhevm-tools-3.5.8-0.1.el6ev.noarch
rhevm-3.5.8-0.1.el6ev.noarch
rhevm-log-collector-3.5.4-2.el6ev.noarch
rhev-guest-tools-iso-3.5-15.el6ev.noarch
rhevm-doc-3.5.3-1.el6eng.noarch
rhevm-spice-client-x86-msi-3.5-3.el6.noarch
rhevm-guest-agent-common-1.0.10-2.el6ev.noarch
rhevm-dwh-setup-3.5.5-1.el6ev.noarch
rhevm-branding-rhev-3.5.0-4.el6ev.noarch
rhevm-cli-3.5.0.6-1.el6ev.noarch
rhevm-dbscripts-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.5.8-0.1.el6ev.noarch
rhevm-reports-setup-3.5.8-1.el6ev.noarch
rhevm-dependencies-3.5.1-1.el6ev.noarch
rhevm-websocket-proxy-3.5.8-0.1.el6ev.noarch
rhevm-userportal-3.5.8-0.1.el6ev.noarch
rhevm-restapi-3.5.8-0.1.el6ev.noarch
Linux version 2.6.32-573.41.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Mar 2 11:08:17 EST 2017
Linux 2.6.32-573.41.1.el6.x86_64 #1 SMP Thu Mar 2 11:08:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)

on pair of hosts:
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
sanlock-3.2.4-3.el7_2.x86_64
rhevm-sdk-python-3.5.6.0-1.el7ev.noarch
mom-0.4.1-4.el7ev.noarch
vdsm-4.16.38-1.el7ev.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.25.x86_64
ovirt-hosted-engine-ha-1.2.10-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.6.x86_64
ovirt-host-deploy-1.3.2-1.el7ev.noarch
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K May  3 13:19 /etc/ovirt-hosted-engine/answers.conf

puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root root 2.6K May  3 13:28 /etc/ovirt-hosted-engine/answers.conf

2)Added 2 NFS data storage domains.
3)Migrated HE-VM to puma19 (second hosted-engine-host) and made it SPM.
4)Upgraded environment to latest 3.6.11, while following https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html and did it on initial hosted-engine-host puma18. I've upgraded both hosts from 3.5el7.2 directly to 3.6el7.3.
5)Got hosted-storage successfully upgraded on both hosts:

puma18 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log | grep 'Successfully'
MainThread::INFO::2017-05-03 17:41:43,778::upgrade::1046::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Successfully upgraded

puma19 ~]#  cat /var/log/ovirt-hosted-engine-ha/agent.log | grep 'Successfully'
MainThread::INFO::2017-05-03 17:06:28,482::upgrade::990::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) Successfully moved the configuration to the shared storage
MainThread::INFO::2017-05-03 17:08:01,744::upgrade::823::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_remove_storage_pool) Successfully removed the shared pool
MainThread::INFO::2017-05-03 17:08:32,557::upgrade::1046::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Successfully upgraded

Works for me on these components on hosts:
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K May  3 13:19 /etc/ovirt-hosted-engine/answers.conf

puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root kvm 2.6K May  3 13:28 /etc/ovirt-hosted-engine/answers.conf
Comment 19 Nikolai Sednev 2017-05-03 14:50:36 UTC 
Components on engine after upgrade:
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.11.1-0.1.el6.noarch
rhevm-sdk-python-3.6.9.1-1.el6ev.noarch
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-setup-base-3.6.11.1-0.1.el6.noarch
rhevm-setup-plugin-websocket-proxy-3.6.11.1-0.1.el6.noarch
rhevm-doc-3.6.10-1.el6ev.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-backend-3.6.11.1-0.1.el6.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch
rhevm-setup-3.6.11.1-0.1.el6.noarch
rhevm-cli-3.6.9.0-1.el6ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el6ev.noarch
rhevm-restapi-3.6.11.1-0.1.el6.noarch
rhevm-dwh-3.6.8-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.7-2.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.6.11.1-0.1.el6.noarch
rhevm-reports-setup-3.6.5.1-1.el6ev.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
rhevm-vmconsole-proxy-helper-3.6.11.1-0.1.el6.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-tools-3.6.11.1-0.1.el6.noarch
rhevm-3.6.11.1-0.1.el6.noarch
rhevm-lib-3.6.11.1-0.1.el6.noarch
rhevm-dwh-setup-3.6.8-1.el6ev.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
rhevm-websocket-proxy-3.6.11.1-0.1.el6.noarch
rhevm-extensions-api-impl-3.6.11.1-0.1.el6.noarch
rhevm-webadmin-portal-3.6.11.1-0.1.el6.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-image-uploader-3.6.1-2.el6ev.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
ovirt-vmconsole-1.0.4-1.el6ev.noarch
rhevm-userportal-3.6.11.1-0.1.el6.noarch
rhevm-reports-3.6.5.1-1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.6.11.1-0.1.el6.noarch
rhevm-dependencies-3.6.1-1.el6ev.noarch
rhevm-tools-backup-3.6.11.1-0.1.el6.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-dbscripts-3.6.11.1-0.1.el6.noarch
Linux version 2.6.32-642.13.2.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Sat Feb 18 16:54:58 EST 2017
Linux 2.6.32-642.13.2.el6.x86_64 #1 SMP Sat Feb 18 16:54:58 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.8 (Santiago)
Comment 21 errata-xmlrpc 2017-05-09 17:05:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1213
Note You need to log in before you can comment on or make changes to this bug.