Bug 1163649 - Error trying to add new hosted-engine host to upgraded Hosted Engine cluster
Summary: Error trying to add new hosted-engine host to upgraded Hosted Engine cluster
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.5.0
Hardware: x86_64
OS: Linux
high
urgent
Target Milestone: ---
: 3.5.0
Assignee: Sandro Bonazzola
QA Contact: Nikolai Sednev
URL:
Whiteboard: integration
Depends On: 1163385
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-13 08:37 UTC by Sandro Bonazzola
Modified: 2015-02-11 20:41 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1163385
Environment:
Last Closed: 2015-02-11 20:41:44 UTC
oVirt Team: ---
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0161 normal SHIPPED_LIVE ovirt-hosted-engine-setup bug fix and enhancement update 2015-12-07 21:35:11 UTC
oVirt gerrit 35104 master MERGED upgrade: allow to add host to updated cluster Never
oVirt gerrit 35137 None None None Never

Description Sandro Bonazzola 2014-11-13 08:37:19 UTC
+++ This bug was initially created as a clone of Bug #1163385 +++

Description of problem:

I have upgraded my oVirt 3.4 hosted engine cluster to oVirt 3.5 using the upgrade instructions on the Wiki.  Everything appears to be working fine after the upgrade.

However, I am now trying to add a new host to the hosted engine configuration but the hosted-engine --deploy fails after sshing the answers file from the upgraded primary configuration.  The following errors can be found in the setup log:


Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host

I confirmed that the answers file on the upgraded host does not have any lockspace UUIDs:

OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
OVEHOSTED_STORAGE/storageType=none:None
OVEHOSTED_STORAGE/volUUID=str:da160775-07fe-4569-b45f-03be0c5896a5
OVEHOSTED_STORAGE/domainType=str:nfs3
OVEHOSTED_STORAGE/imgSizeGB=str:25
OVEHOSTED_STORAGE/storageDomainConnection=str:192.168.8.12:/mnt/data2/vm/engine
OVEHOSTED_STORAGE/connectionUUID=str:880093ea-b0c1-448d-ac55-cde99feebc23
OVEHOSTED_STORAGE/spUUID=str:5e7ff7c2-6e75-4ba8-a5cc-e8dc5d37e478
OVEHOSTED_STORAGE/imgUUID=str:c9466bb6-a78c-4caa-bce3-22c87a5f3f1a
OVEHOSTED_STORAGE/sdUUID=str:b12fd59c-380a-40b3-b7f2-02d455de1d3b

Is there something I can do to update the answers file on the updated 3.5 working host so this will work?

As an aside the new host is EL7 whereas the existing hosts are EL6.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.2.1-1.el7.noarch

How reproducible:


Steps to Reproduce:
1. Install an oVirt 3.4 hosted-engine environment.
2. Upgrade environment to 3.5 using information in Wiki.
3. Attempt to add new hosted-engine host using hosted-engine --deploy

Actual results:
Adding new host fails with error:

Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host

Expected results:
Able to add new host to hosted-engine cluster.

Additional info:

Here is the relevant section of the hosted-engine setup log file:

2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage validation METHOD otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace.Plugin._validation
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/sanlock/lockspace.py", line 102, in _validation
    'Answer file lacks lockspace UUIDs, please use an '
RuntimeError: Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host
2014-11-11 22:57:04 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host',), <traceback object at 0x34c85a8>)]'
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2014-11-11 22:57:04 INFO otopi.context context.runSequence:417 Stage: Clean up
2014-11-11 22:57:04 DEBUG otopi.context context.runSequence:421 STAGE cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.engine.add_host.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:609 spmStop
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._cleanup:970 Not SPM?
Traceback (most recent call last):
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 968, in _cleanup
    self._spmStop()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 617, in _spmStop
    raise RuntimeError(status_uuid)
RuntimeError: Not SPM
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:580 disconnectStoragePool
2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._cleanup
2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin._save_answers_at_cleanup
2014-11-11 22:57:08 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers:48 Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
2014-11-11 22:57:09 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers_at_cleanup:104 Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated

--- Additional comment from Jiri Moskovcak on 2014-11-13 03:21:52 EST ---

In case of updating from 3.4 we need to use FileSystemBackend to handle the storage init while adding new host. 

idea:
if the answer file lacks lockspace uuid and the storage tyupe is nfs we can assume it's 3.4 answer file, so we should use FileSystemBackend

Comment 2 Nikolai Sednev 2015-01-13 12:24:22 UTC
Works for me on these components:
RHEL7.0 was added to setup after upgraded to latest vt:
vdsm-4.16.8.1-5.el7ev.x86_64
sanlock-3.1.0-2.el7.x86_64
libvirt-client-1.1.1-29.el7_0.4.x86_64
mom-0.4.1-4.el7ev.noarch
qemu-kvm-rhev-1.5.3-60.el7_0.11.x86_64

RHEL6.6 host, which was hosting the HE:
vdsm-4.16.8.1-5.el6ev.x86_64
sanlock-2.8-1.el6.x86_64
libvirt-client-0.10.2-46.el6_6.2.x86_64
qemu-kvm-rhev-0.12.1.2-2.448.el6.x86_64
mom-0.4.1-4.el6ev.noarch

Engine's lates vt:
rhevm-3.5.0-0.28.el6ev.noarch


Of course RHEL7.0 host can't join HE host cluster and within engine error being received "2015-Jan-13, 10:15 Not possible to mix RHEL 6.x and 7.x hosts in one cluster. Tried adding RHEL - 7.0 - 1.el7_0.1 host to a cluster with RHEL - 6Server - 6.6.0.2.el6 hosts."
And host becomes non-operational.

In case of RHEL6.6 being added, no problems appear and it becomes active within the WEBUI.

Comment 4 errata-xmlrpc 2015-02-11 20:41:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0161.html


Note You need to log in before you can comment on or make changes to this bug.