Bug 1319595 - ha-agent not starting when upgrading the hosted engine from 6.x to 7.x
Summary: ha-agent not starting when upgrading the hosted engine from 6.x to 7.x
Keywords:
Status: CLOSED DUPLICATE of bug 1328382
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.6.3
Hardware: All
OS: Linux
high
medium
Target Milestone: ovirt-3.6.6
: ---
Assignee: Simone Tiraboschi
QA Contact: Artyom
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-21 07:39 UTC by nijin ashok
Modified: 2016-07-19 09:07 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-18 09:46:17 UTC
oVirt Team: Integration
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1316143 0 urgent CLOSED 3.6 hosted-engine hosts can't be added properly to 3.6 host cluster that was started with 3.4. 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1328382 0 urgent CLOSED [DOC] Hosted engine upgrade guide should include the supportability of direct upgrade from 3.5 el6 to 3.6 el7 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1337641 0 high CLOSED Special procedure needed to upgrade HE environment with RHEL6 hosts to RHEL7 2021-02-22 00:41:40 UTC

Internal Links: 1316143 1328382 1337641

Description nijin ashok 2016-03-21 07:39:17 UTC
Description of problem:

Following this article to upgrade 3.5 hosted-engine with 6.7 hosts to 3.6.

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Self-Hosted_Engine_Guide/index.html#Upgrading_the_Self-Hosted_Engine_from_6_to_7

1. Remove one of the host from RHEV-M
2. Create a new cluster for 7.x host
3. Reinstalled this hypervisor with Red Hat Enterprise Virtualization Hypervisor release 7.2 (20160302.1.el7ev)
4. Re-deploy the host with hosted-engine --deploy and used the same host id 2
5. Change the host into new cluster so that the host become active.

However the ha-agent service in the 7.2 host is not coming up and can see below error in the agent.log.

MainThread::ERROR::2016-03-20 09:51:01,831::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Path to volume c23122fe-9b27-4ef6-b7b8-8816df43ae90 not found in /rhev/data-center/mnt' - trying to restart agent
MainThread::WARNING::2016-03-20 09:51:06,837::agent::208::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'

c23122fe-9b27-4ef6-b7b8-8816df43ae90 is conf_volume_UUID from the hosted-engine.conf file. This lv c23122fe-9b27-4ef6-b7b8-8816df43ae90 is not created too. However reinstalling the host with rhevh-7.2-20160219.0.el6ev.iso (latest version before 3.6) and upgrading to rhevh-7.2-20160302.1.el6ev.iso seems to be working good. conf_volume_UUID is also created.

MainThread::INFO::2016-03-20 08:29:15,800::upgrade::151::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Looking for conf volume
MainThread::ERROR::2016-03-20 08:29:15,918::upgrade::207::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_is_conf_volume_there) Unable to find HE conf volume
MainThread::INFO::2016-03-20 08:29:15,919::upgrade::262::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(_create_shared_conf_volume) Creating hosted-engine configuration volume on the shared storage domain


Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.3.3.4-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.4.3-1.el7ev.noarch


How reproducible:
100%

Steps to Reproduce:
1. Add a new rhevh-7.2-20160302.1.el6ev host to an existing 3.5 hosted engine setup.
2. ha-agent is not starting with below error.

MainThread::ERROR::2016-03-20 09:51:01,831::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Path to volume c23122fe-9b27-4ef6-b7b8-8816df43ae90 not found in /rhev/data-center/mnt' - trying to restart agent



Actual results:

ha-agent is not starting

Expected results:

ha-agent should work.

Additional info:

Comment 1 Simone Tiraboschi 2016-03-22 14:46:37 UTC
Nijin, can you please retry with the next (today ?) rhev-h build which should include ovirt-ha-agente-1.3.5.1 ?
I strongly suspect it's just a dusplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1316143

Comment 2 Sandro Bonazzola 2016-04-06 11:09:35 UTC
Closing the bug with insufficient data.
Please reopen if you can provide needed info.

Comment 3 nijin ashok 2016-04-09 09:46:09 UTC
Sorry for the delay.

Reopening as I was able to replicate the issue in latest ha-agent .

Tested with below version

vdsm-4.17.23.2-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.5.1-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.4.0-1.el7ev.noarch

MainThread::WARNING::2016-04-09 07:40:57,783::ovf_store::104::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE
MainThread::ERROR::2016-04-09 07:40:57,783::config::234::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf
MainThread::ERROR::2016-04-09 07:40:57,784::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Error: 'Path to volume None not found in /rhev/data-center/mnt' - trying to restart agent
MainThread::WARNING::2016-04-09 07:41:02,788::agent::208::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '5'


The conf_volume value is none.

grep "conf" /etc/ovirt-hosted-engine/hosted-engine.conf 
conf=/var/run/ovirt-hosted-engine-ha/vm.conf
conf_volume_UUID=None
conf_image_UUID=None


As earlier mentioned, it works when we upgrade from 3.5 el6 version to the 3.5 el7 version as tested in https://bugzilla.redhat.com/show_bug.cgi?id=1306825#c14 . But fails when we directly upgrade from 3.5 el6 version to 3.6 el7 version.

Comment 4 Sandro Bonazzola 2016-04-18 09:46:17 UTC
(In reply to nijin ashok from comment #3)

> 
> As earlier mentioned, it works when we upgrade from 3.5 el6 version to the
> 3.5 el7 version as tested in
> https://bugzilla.redhat.com/show_bug.cgi?id=1306825#c14 . But fails when we
> directly upgrade from 3.5 el6 version to 3.6 el7 version.

So, closing as not a bug since the direct upgrade from 3.5 el6 to 3.6 el7 is not supported.
As reported, 3.5 el6 to 3.5 el7 and then upgrade to el7 is the proper way to perform the upgrade.

Comment 5 nijin ashok 2016-04-21 15:25:43 UTC
(In reply to Sandro Bonazzola from comment #4)
> (In reply to nijin ashok from comment #3)
> 
> > 
> > As earlier mentioned, it works when we upgrade from 3.5 el6 version to the
> > 3.5 el7 version as tested in
> > https://bugzilla.redhat.com/show_bug.cgi?id=1306825#c14 . But fails when we
> > directly upgrade from 3.5 el6 version to 3.6 el7 version.
> 
> So, closing as not a bug since the direct upgrade from 3.5 el6 to 3.6 el7 is
> not supported.
> As reported, 3.5 el6 to 3.5 el7 and then upgrade to el7 is the proper way to
> perform the upgrade.

So what will be the case of RHEL host? How we can restrict the vdsm version to the one which is released with 3.5? Also can you please check this doc bug https://bugzilla.redhat.com/show_bug.cgi?id=1328382 .

Comment 6 Sandro Bonazzola 2016-05-04 12:57:53 UTC
(In reply to nijin ashok from comment #5)
> So what will be the case of RHEL host?

Not sure to have understood the question. Hosted engine is supported in 3.5 on both EL6 and EL7 so the upgrade path is still:
3.5 EL6 -> 3.5 EL7 -> 3.6 EL7

> How we can restrict the vdsm version
> to the one which is released with 3.5?


when installing 3.5 el7 just don't subscribe to the 3.6 channel, that should allow to install clean 3.5 el7. Once it's done, enable 3.6 channel and perform the upgrade.

> Also can you please check this doc
> bug https://bugzilla.redhat.com/show_bug.cgi?id=1328382 .

Done.

Comment 7 nijin ashok 2016-05-05 17:31:06 UTC
(In reply to Sandro Bonazzola from comment #6)
> (In reply to nijin ashok from comment #5)
> when installing 3.5 el7 just don't subscribe to the 3.6 channel, that should
> allow to install clean 3.5 el7. Once it's done, enable 3.6 channel and
> perform the upgrade.

Got it. But in RHEL 7, I think we don't have any version specific channel for hosted-engine packages. For both 3.5 and 3.6, it comes from  rhel-7-server-rhev-mgmt-agent-rpms 

ovirt-hosted-engine-ha-1.2.4-5.el7ev.noarch : RHEV-M Hosted Engine High Availability Manager
Repo        : rhel-7-server-rhev-mgmt-agent-rpms

ovirt-hosted-engine-ha-1.3.5.3-1.el7ev.noarch : RHEV-M Hosted Engine High Availability Manager
Repo        : @rhel-7-server-rhev-mgmt-agent-rpms

3.5 Doc : https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.5/html-single/Installation_Guide/index.html#Subscribing_to_the_Required_Entitlements

3.6 Doc : https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.6/html-single/Self-Hosted_Engine_Guide/index.html#Subscribing_to_the_Required_Entitlements

> > Also can you please check this doc
> > bug https://bugzilla.redhat.com/show_bug.cgi?id=1328382 .
> 
> Done.

Thanks!

Comment 8 Sandro Bonazzola 2016-07-19 08:44:37 UTC
Ok, understood the issue.
See also bug #1337641 and bug #1328382 with regards to the upgrade procedure for 3.5 -> 3.6 with Hosted Engine.

Closing this as duplicate of bug #1328382

*** This bug has been marked as a duplicate of bug 1328382 ***


Note You need to log in before you can comment on or make changes to this bug.