Bug 1422962

Summary:

[downstream clone - 3.6.11] [Hosted-Engine] 3.5 HE SD upgrade fails if done on initial host

Product:

Red Hat Enterprise Virtualization Manager

Reporter:

rhev-integ

Component:

ovirt-hosted-engine-ha

Assignee:

Simone Tiraboschi <stirabos>

Status:

CLOSED ERRATA

QA Contact:

Nikolai Sednev <nsednev>

Severity:

high

Docs Contact:

Priority:

high

Version:

3.6.9

CC:

eedri, lsurette, mavital, melewis, nsednev, pbrilla, stirabos, ykaul, ylavi

Target Milestone:

ovirt-3.6.11

Keywords:

Triaged, ZStream

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch

Doc Type:

Bug Fix

Doc Text:

With this update, incorrect permissions causing the Red Hat Enterprise Virtualization 3.5 self-hosted engine storage domain upgrade to fail if it was performed on the initial host have been corrected.

Story Points:

---

Clone Of:

1422864

Environment:

Last Closed:

2017-05-09 17:05:03 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

Integration

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1422864

Bug Blocks:

1400800, 1434954

Attachments:

Description	Flags
engine	none
sosreport-puma18	none
sosreport-puma19	none

Description rhev-integ 2017-02-16 16:37:45 UTC

+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1422864 +++
======================================================================

+++ This bug is a downstream clone. The original bug is: +++
+++   bug 1401359 +++
======================================================================

Description of problem:

In 3.5, /etc/ovirt-hosted-engine/answers.conf permissions are, right after install, as below:

Initial Host:
-rw-rw----. 1 root root 2585 Dec  5 00:08 /etc/ovirt-hosted-engine/answers.conf

Additional Hosts:
-rw-r--r--. 1 root root 2575 Dec  5 00:18 /etc/ovirt-hosted-engine/answers.conf

When upgrading to 3.6, if the Host chosen to upgrade first (to ha 1.3.x) is the Initial one selected for the initial deployment of HE (-rw-rw----), the HE SD upgrade fails due to EACCESS to answers.conf file.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160920.1.el7ev)
ovirt-hosted-engine-ha-1.3.5.8-1.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy fresh Hosted Engine on 3.5 using 20160219.0.el7ev
2. Upgrade initial HE host to 20160920.1.el7ev
3. Trigger HE SD Upgrade (Host in maintenance, restart ha-agent)

Actual results:
If one chooses the initial HE Host to do the upgrade, the HE SD not Upgraded, ha-agent keeps restarting. See:
MainThread::INFO::2016-12-05 01:24:38,917::upgrade::1010::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Upgrading to current version
MainThread::INFO::2016-12-05 01:24:39,004::upgrade::736::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_stopMonitoringDomain) Stop monitoring domain
MainThread::INFO::2016-12-05 01:24:39,059::upgrade::151::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Looking for conf volume
MainThread::ERROR::2016-12-05 01:24:39,112::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume
MainThread::INFO::2016-12-05 01:24:39,112::upgrade::953::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) _move_to_shared_conf
MainThread::INFO::2016-12-05 01:24:39,112::upgrade::375::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Reading conf file: fhanswers.conf
MainThread::ERROR::2016-12-05 01:24:39,112::upgrade::399::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2016-12-05 01:24:39,113::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent

Expected results:
HE SD upgraded. 

Additional info:
The permission issue seem to be there for some time. In previous hosted-engine-ha in 3.6 apparently used to upgrade the HE SD even when hitting this "Permission denied" error. See: https://bugzilla.redhat.com/show_bug.cgi?id=1292652#c1

Apparently we missed that error in that BZ, and now the behavior is slightly different, we restart the agent and the HE SD is NOT upgraded at the step it should. This causes troubles for 3.5 to 3.6 Upgrade.

(Originally by Germano Veit Michel)

(Originally by rhev-integ)

Comment 6 Nikolai Sednev 2017-04-12 20:38:43 UTC

1)Deployed clean environment on puma18 and puma19 hosts, over NFS storage domain and added two NFS data storage domains.
Components on hosts:
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
sanlock-3.2.4-3.el7_2.x86_64
rhevm-sdk-python-3.5.6.0-1.el7ev.noarch
mom-0.4.1-4.el7ev.noarch
vdsm-4.16.38-1.el7ev.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.25.x86_64
ovirt-hosted-engine-ha-1.2.10-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.6.x86_64
ovirt-host-deploy-1.3.2-1.el7ev.noarch
Linux version 3.10.0-327.53.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Tue Mar 14 10:49:09 EDT 2017
Linux 3.10.0-327.53.1.el7.x86_64 #1 SMP Tue Mar 14 10:49:09 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

On engine:
rhevm-lib-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.5.8-0.1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch
rhevm-setup-plugins-3.5.4-1.el6ev.noarch
rhevm-iso-uploader-3.5.1-1.el6ev.noarch
rhevm-extensions-api-impl-3.5.8-0.1.el6ev.noarch
rhevm-spice-client-x64-msi-3.5-3.el6.noarch
rhevm-backend-3.5.8-0.1.el6ev.noarch
rhevm-sdk-python-3.5.6.0-1.el6ev.noarch
ovirt-host-deploy-1.3.2-1.el6ev.noarch
rhevm-spice-client-x64-cab-3.5-3.el6.noarch
rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch
rhevm-setup-base-3.5.8-0.1.el6ev.noarch
rhevm-reports-3.5.8-1.el6ev.noarch
rhevm-image-uploader-3.5.0-4.el6ev.noarch
rhevm-setup-plugin-websocket-proxy-3.5.8-0.1.el6ev.noarch
ovirt-host-deploy-java-1.3.2-1.el6ev.noarch
rhevm-spice-client-x86-cab-3.5-3.el6.noarch
rhevm-setup-3.5.8-0.1.el6ev.noarch
rhevm-tools-3.5.8-0.1.el6ev.noarch
rhevm-3.5.8-0.1.el6ev.noarch
rhevm-log-collector-3.5.4-2.el6ev.noarch
rhev-guest-tools-iso-3.5-15.el6ev.noarch
rhevm-doc-3.5.3-1.el6eng.noarch
rhevm-spice-client-x86-msi-3.5-3.el6.noarch
rhevm-guest-agent-common-1.0.10-2.el6ev.noarch
rhevm-dwh-setup-3.5.5-1.el6ev.noarch
rhevm-branding-rhev-3.5.0-4.el6ev.noarch
rhevm-cli-3.5.0.6-1.el6ev.noarch
rhevm-dbscripts-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.5.8-0.1.el6ev.noarch
rhevm-reports-setup-3.5.8-1.el6ev.noarch
rhevm-dependencies-3.5.1-1.el6ev.noarch
rhevm-websocket-proxy-3.5.8-0.1.el6ev.noarch
rhevm-userportal-3.5.8-0.1.el6ev.noarch
rhevm-restapi-3.5.8-0.1.el6ev.noarch
Linux version 2.6.32-573.41.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Mar 2 11:08:17 EST 2017
Linux 2.6.32-573.41.1.el6.x86_64 #1 SMP Thu Mar 2 11:08:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)

puma18 ~]# hosted-engine --vm-status


--== Host 1 status ==--

Status up-to-date                  : True
Hostname                           : puma18
Host ID                            : 1
Engine status                      : {"health": "good", "vm": "up", "detail": "up"}
Score                              : 2400
Local maintenance                  : False
Host timestamp                     : 21579
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=21579 (Wed Apr 12 19:11:01 2017)
        host-id=1
        score=2400
        maintenance=False
        state=EngineUp


--== Host 2 status ==--

Status up-to-date                  : True
Hostname                           : puma19
Host ID                            : 2
Engine status                      : {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}
Score                              : 2400
Local maintenance                  : False
Host timestamp                     : 21438
Extra metadata (valid at timestamp):
        metadata_parse_version=1
        metadata_feature_version=1
        timestamp=21438 (Wed Apr 12 19:11:02 2017)
        host-id=2
        score=2400
        maintenance=False
        state=EngineDown


2)Permissions right after deployment on both hosts as appears bellow:
puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K Apr 12 17:52 /etc/ovirt-hosted-engine/answers.conf
puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root root 2.6K Apr 12 18:01 /etc/ovirt-hosted-engine/answers.conf

3)Created one guest VM with RHEL7.3 OS.
4)Upgraded HE from 3.5 to 3.6 following https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html
5)I was able to reproduce this issue on my environment.
MainThread::INFO::2017-04-12 22:24:36,343::upgrade::1012::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Upgrading to current version
MainThread::INFO::2017-04-12 22:24:36,368::upgrade::738::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_stopMonitoringDomain) Stop monitoring domain
MainThread::INFO::2017-04-12 22:24:36,383::upgrade::153::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Looking for conf volume
MainThread::ERROR::2017-04-12 22:24:36,399::upgrade::209::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume
MainThread::INFO::2017-04-12 22:24:36,399::upgrade::955::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) _move_to_shared_conf
MainThread::INFO::2017-04-12 22:24:36,400::upgrade::377::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Reading conf file: fhanswers.conf
MainThread::ERROR::2017-04-12 22:24:36,400::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-12 22:24:36,400::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent

Moving back to assigned.

I was not able to move the second host to maintenance as HE-VM was running on it and it was failing to migrate to 3.6's rhel7.3 puma18.

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K Apr 12 17:52 /etc/ovirt-hosted-engine/answers.conf
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
Linux version 3.10.0-514.16.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Fri Mar 10 13:12:32 EST 2017
Linux puma18 3.10.0-514.16.1.el7.x86_64 #1 SMP Fri Mar 10 13:12:32 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Comment 7 Nikolai Sednev 2017-04-12 20:39:44 UTC

Created attachment 1271264 [details]
engine

Comment 8 Nikolai Sednev 2017-04-12 20:41:06 UTC

Created attachment 1271265 [details]
sosreport-puma18

Comment 9 Nikolai Sednev 2017-04-12 20:42:17 UTC

Created attachment 1271266 [details]
sosreport-puma19

Comment 10 Simone Tiraboschi 2017-04-13 09:53:38 UTC

ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch didn't included the fix, I built ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch with it

Comment 11 Nikolai Sednev 2017-04-25 15:48:35 UTC

I still don't see ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch being distributed to latest 3.6.

Still being reproduced on latest 3.6 components on host:
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-1.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)


'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent
MainThread::ERROR::2017-04-25 18:41:40,260::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-25 18:41:40,260::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent
MainThread::ERROR::2017-04-25 18:41:46,008::upgrade::401::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_get_conffile_content) Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'
MainThread::ERROR::2017-04-25 18:41:46,008::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Failed to read configuration file '/etc/ovirt-hosted-engine/answers.conf': [Errno 13] Permission denied: '/etc/ovirt-hosted-engine/answers.conf'' - trying to restart agent

Comment 18 Nikolai Sednev 2017-05-03 14:47:05 UTC

1)
Deployed clean 3.5.8-0.1.el6ev hosted engine over NFS:
rhevm-lib-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.5.8-0.1.el6ev.noarch
rhevm-dwh-3.5.5-1.el6ev.noarch
rhevm-setup-plugins-3.5.4-1.el6ev.noarch
rhevm-iso-uploader-3.5.1-1.el6ev.noarch
rhevm-extensions-api-impl-3.5.8-0.1.el6ev.noarch
rhevm-spice-client-x64-msi-3.5-3.el6.noarch
rhevm-backend-3.5.8-0.1.el6ev.noarch
rhevm-sdk-python-3.5.6.0-1.el6ev.noarch
ovirt-host-deploy-1.3.2-1.el6ev.noarch
rhevm-spice-client-x64-cab-3.5-3.el6.noarch
rhevm-webadmin-portal-3.5.8-0.1.el6ev.noarch
rhevm-setup-base-3.5.8-0.1.el6ev.noarch
rhevm-reports-3.5.8-1.el6ev.noarch
rhevm-image-uploader-3.5.0-4.el6ev.noarch
rhevm-setup-plugin-websocket-proxy-3.5.8-0.1.el6ev.noarch
ovirt-host-deploy-java-1.3.2-1.el6ev.noarch
rhevm-spice-client-x86-cab-3.5-3.el6.noarch
rhevm-setup-3.5.8-0.1.el6ev.noarch
rhevm-tools-3.5.8-0.1.el6ev.noarch
rhevm-3.5.8-0.1.el6ev.noarch
rhevm-log-collector-3.5.4-2.el6ev.noarch
rhev-guest-tools-iso-3.5-15.el6ev.noarch
rhevm-doc-3.5.3-1.el6eng.noarch
rhevm-spice-client-x86-msi-3.5-3.el6.noarch
rhevm-guest-agent-common-1.0.10-2.el6ev.noarch
rhevm-dwh-setup-3.5.5-1.el6ev.noarch
rhevm-branding-rhev-3.5.0-4.el6ev.noarch
rhevm-cli-3.5.0.6-1.el6ev.noarch
rhevm-dbscripts-3.5.8-0.1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.5.8-0.1.el6ev.noarch
rhevm-reports-setup-3.5.8-1.el6ev.noarch
rhevm-dependencies-3.5.1-1.el6ev.noarch
rhevm-websocket-proxy-3.5.8-0.1.el6ev.noarch
rhevm-userportal-3.5.8-0.1.el6ev.noarch
rhevm-restapi-3.5.8-0.1.el6ev.noarch
Linux version 2.6.32-573.41.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Thu Mar 2 11:08:17 EST 2017
Linux 2.6.32-573.41.1.el6.x86_64 #1 SMP Thu Mar 2 11:08:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)

on pair of hosts:
ovirt-hosted-engine-setup-1.2.6.1-1.el7ev.noarch
sanlock-3.2.4-3.el7_2.x86_64
rhevm-sdk-python-3.5.6.0-1.el7ev.noarch
mom-0.4.1-4.el7ev.noarch
vdsm-4.16.38-1.el7ev.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.25.x86_64
ovirt-hosted-engine-ha-1.2.10-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.6.x86_64
ovirt-host-deploy-1.3.2-1.el7ev.noarch
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K May  3 13:19 /etc/ovirt-hosted-engine/answers.conf

puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root root 2.6K May  3 13:28 /etc/ovirt-hosted-engine/answers.conf

2)Added 2 NFS data storage domains.
3)Migrated HE-VM to puma19 (second hosted-engine-host) and made it SPM.
4)Upgraded environment to latest 3.6.11, while following https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html/Self-Hosted_Engine_Guide/Upgrading_the_Self-Hosted_Engine.html and did it on initial hosted-engine-host puma18. I've upgraded both hosts from 3.5el7.2 directly to 3.6el7.3.
5)Got hosted-storage successfully upgraded on both hosts:

puma18 ~]# cat /var/log/ovirt-hosted-engine-ha/agent.log | grep 'Successfully'
MainThread::INFO::2017-05-03 17:41:43,778::upgrade::1046::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Successfully upgraded

puma19 ~]#  cat /var/log/ovirt-hosted-engine-ha/agent.log | grep 'Successfully'
MainThread::INFO::2017-05-03 17:06:28,482::upgrade::990::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_move_to_shared_conf) Successfully moved the configuration to the shared storage
MainThread::INFO::2017-05-03 17:08:01,744::upgrade::823::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_remove_storage_pool) Successfully removed the shared pool
MainThread::INFO::2017-05-03 17:08:32,557::upgrade::1046::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Successfully upgraded

Works for me on these components on hosts:
rhevm-sdk-python-3.6.9.1-1.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.10-2.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
sanlock-3.4.0-1.el7.x86_64
mom-0.5.6-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.7.4-1.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
vdsm-4.17.39-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-2.0.0-10.el7_3.5.x86_64
qemu-kvm-rhev-2.6.0-28.el7_3.9.x86_64
Linux version 3.10.0-327.55.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Fri Apr 21 02:14:48 EDT 2017
Linux 3.10.0-327.55.1.el7.x86_64 #1 SMP Fri Apr 21 02:14:48 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.3 (Maipo)

puma18 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-rw----. 1 root root 2.6K May  3 13:19 /etc/ovirt-hosted-engine/answers.conf

puma19 ~]# ll -lsha /etc/ovirt-hosted-engine/answers.conf
4.0K -rw-r--r--. 1 root kvm 2.6K May  3 13:28 /etc/ovirt-hosted-engine/answers.conf

Comment 19 Nikolai Sednev 2017-05-03 14:50:36 UTC

Components on engine after upgrade:
rhevm-setup-plugin-vmconsole-proxy-helper-3.6.11.1-0.1.el6.noarch
rhevm-sdk-python-3.6.9.1-1.el6ev.noarch
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
rhevm-spice-client-x64-msi-3.6-7.el6.noarch
rhevm-setup-base-3.6.11.1-0.1.el6.noarch
rhevm-setup-plugin-websocket-proxy-3.6.11.1-0.1.el6.noarch
rhevm-doc-3.6.10-1.el6ev.noarch
rhevm-spice-client-x86-msi-3.6-7.el6.noarch
rhevm-backend-3.6.11.1-0.1.el6.noarch
rhevm-guest-agent-common-1.0.11-6.el6ev.noarch
ovirt-setup-lib-1.0.1-1.el6ev.noarch
rhevm-setup-3.6.11.1-0.1.el6.noarch
rhevm-cli-3.6.9.0-1.el6ev.noarch
ovirt-vmconsole-proxy-1.0.4-1.el6ev.noarch
rhevm-restapi-3.6.11.1-0.1.el6.noarch
rhevm-dwh-3.6.8-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.7-2.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-3.6.11.1-0.1.el6.noarch
rhevm-reports-setup-3.6.5.1-1.el6ev.noarch
rhevm-iso-uploader-3.6.0-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
rhevm-vmconsole-proxy-helper-3.6.11.1-0.1.el6.noarch
rhevm-spice-client-x64-cab-3.6-7.el6.noarch
rhevm-tools-3.6.11.1-0.1.el6.noarch
rhevm-3.6.11.1-0.1.el6.noarch
rhevm-lib-3.6.11.1-0.1.el6.noarch
rhevm-dwh-setup-3.6.8-1.el6ev.noarch
rhevm-log-collector-3.6.1-1.el6ev.noarch
rhevm-websocket-proxy-3.6.11.1-0.1.el6.noarch
rhevm-extensions-api-impl-3.6.11.1-0.1.el6.noarch
rhevm-webadmin-portal-3.6.11.1-0.1.el6.noarch
rhev-guest-tools-iso-3.6-6.el6ev.noarch
rhevm-setup-plugins-3.6.5-1.el6ev.noarch
rhevm-image-uploader-3.6.1-2.el6ev.noarch
rhevm-branding-rhev-3.6.0-10.el6ev.noarch
ovirt-vmconsole-1.0.4-1.el6ev.noarch
rhevm-userportal-3.6.11.1-0.1.el6.noarch
rhevm-reports-3.6.5.1-1.el6ev.noarch
rhevm-setup-plugin-ovirt-engine-common-3.6.11.1-0.1.el6.noarch
rhevm-dependencies-3.6.1-1.el6ev.noarch
rhevm-tools-backup-3.6.11.1-0.1.el6.noarch
rhevm-spice-client-x86-cab-3.6-7.el6.noarch
rhevm-dbscripts-3.6.11.1-0.1.el6.noarch
Linux version 2.6.32-642.13.2.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC) ) #1 SMP Sat Feb 18 16:54:58 EST 2017
Linux 2.6.32-642.13.2.el6.x86_64 #1 SMP Sat Feb 18 16:54:58 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.8 (Santiago)

Comment 21 errata-xmlrpc 2017-05-09 17:05:03 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1213