Bug 1163385

Summary: Error trying to add new hosted-engine host to upgraded Hosted Engine cluster
Product: [Retired] oVirt Reporter: David King <david>
Component: ovirt-hosted-engine-setupAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED CURRENTRELEASE QA Contact: Artyom <alukiano>
Severity: urgent Docs Contact:
Priority: high    
Version: 3.5CC: alukiano, daniel.helgenberger, david, didi, ecohen, gklein, iheim, jmoskovc, lsurette, lveyde, msivak, rbalakri, sbonazzo, stirabos, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.5.1Flags: sbonazzo: ovirt_requires_release_note-
Hardware: x86_64   
OS: Linux   
Whiteboard: integration
Fixed In Version: ovirt-3.5.1_rc1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1163649 (view as bug list) Environment:
Last Closed: 2015-01-21 16:04:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1155170, 1163649, 1164308, 1164311, 1192937    
Attachments:
Description Flags
logs and answer file none

Description David King 2014-11-12 15:32:10 UTC
Description of problem:

I have upgraded my oVirt 3.4 hosted engine cluster to oVirt 3.5 using the upgrade instructions on the Wiki.  Everything appears to be working fine after the upgrade.

However, I am now trying to add a new host to the hosted engine configuration but the hosted-engine --deploy fails after sshing the answers file from the upgraded primary configuration.  The following errors can be found in the setup log:


Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host

I confirmed that the answers file on the upgraded host does not have any lockspace UUIDs:

OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter
OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage
OVEHOSTED_STORAGE/storageType=none:None
OVEHOSTED_STORAGE/volUUID=str:da160775-07fe-4569-b45f-03be0c5896a5
OVEHOSTED_STORAGE/domainType=str:nfs3
OVEHOSTED_STORAGE/imgSizeGB=str:25
OVEHOSTED_STORAGE/storageDomainConnection=str:192.168.8.12:/mnt/data2/vm/engine
OVEHOSTED_STORAGE/connectionUUID=str:880093ea-b0c1-448d-ac55-cde99feebc23
OVEHOSTED_STORAGE/spUUID=str:5e7ff7c2-6e75-4ba8-a5cc-e8dc5d37e478
OVEHOSTED_STORAGE/imgUUID=str:c9466bb6-a78c-4caa-bce3-22c87a5f3f1a
OVEHOSTED_STORAGE/sdUUID=str:b12fd59c-380a-40b3-b7f2-02d455de1d3b

Is there something I can do to update the answers file on the updated 3.5 working host so this will work?

As an aside the new host is EL7 whereas the existing hosts are EL6.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup-1.2.1-1.el7.noarch

How reproducible:


Steps to Reproduce:
1. Install an oVirt 3.4 hosted-engine environment.
2. Upgrade environment to 3.5 using information in Wiki.
3. Attempt to add new hosted-engine host using hosted-engine --deploy

Actual results:
Adding new host fails with error:

Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host

Expected results:
Able to add new host to hosted-engine cluster.

Additional info:

Here is the relevant section of the hosted-engine setup log file:

2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage validation METHOD otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace.Plugin._validation
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/sanlock/lockspace.py", line 102, in _validation
    'Answer file lacks lockspace UUIDs, please use an '
RuntimeError: Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host
2014-11-11 22:57:04 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host',), <traceback object at 0x34c85a8>)]'
2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2014-11-11 22:57:04 INFO otopi.context context.runSequence:417 Stage: Clean up
2014-11-11 22:57:04 DEBUG otopi.context context.runSequence:421 STAGE cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.engine.add_host.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._cleanup
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:609 spmStop
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._cleanup:970 Not SPM?
Traceback (most recent call last):
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 968, in _cleanup
    self._spmStop()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 617, in _spmStop
    raise RuntimeError(status_uuid)
RuntimeError: Not SPM
2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:580 disconnectStoragePool
2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._cleanup
2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin._save_answers_at_cleanup
2014-11-11 22:57:08 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers:48 Generating answer file '/etc/ovirt-hosted-engine/answers.conf'
2014-11-11 22:57:09 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers_at_cleanup:104 Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated

Comment 1 Jiri Moskovcak 2014-11-13 08:21:52 UTC
In case of updating from 3.4 we need to use FileSystemBackend to handle the storage init while adding new host. 

idea:
if the answer file lacks lockspace uuid and the storage tyupe is nfs we can assume it's 3.4 answer file, so we should use FileSystemBackend

Comment 2 Artyom 2014-12-04 16:50:14 UTC
I see that fix already in package ovirt-hosted-engine-setup-1.2.1-7.el6ev.noarch,
and setup stage of adding hosts run ok, but after this broker failed to connect to storage:

Thread-481::ERROR::2014-12-04 18:43:59,291::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error handling request, data: 'set-storage-domain VdsmBackend hosted-engine.lockspace=7B22696D6167655F75756964223A20224E6F6E65222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A20224E6F6E65227D sp_uuid=3d30c5d1-a76c-4c79-a6ee-5d92a638e623 dom_type=nfs3 hosted-engine.metadata=7B22696D6167655F75756964223A20224E6F6E65222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A20224E6F6E65227D sd_uuid=d67ca20f-a4be-4b3e-9a96-78474491111f'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle
    data)
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch
    .set_storage_domain(client, sd_type, **options)
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain
    self._backends[client].connect()
  File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 370, in connect
    raise RuntimeError(response["status"]["message"])
RuntimeError: Volume does not exist: ('None',)
I looks like because path, broker now receive some incorrect storage details.
So what you prefer, to open new bug or just continue work with this one.

Anyway I will add answer file, broker, vdsm and agent logs.

Comment 3 Artyom 2014-12-04 16:50:47 UTC
Created attachment 964755 [details]
logs and answer file

Comment 4 Jiri Moskovcak 2014-12-05 08:34:36 UTC
Artyom, can you please also provide hosted-engine.conf. From the code it looks like setup writes the 'None' as the volume_uuid and agent tries to use it as a real value. 

Sandro, can you make setup to not write that value to config if it's None?

Comment 5 Sandro Bonazzola 2014-12-05 08:56:53 UTC
(In reply to Artyom from comment #2)

> So what you prefer, to open new bug or just continue work with this one.
> 
> Anyway I will add answer file, broker, vdsm and agent logs.

I prefer a new BZ.

(In reply to Jiri Moskovcak from comment #4)
> Sandro, can you make setup to not write that value to config if it's None?

1) not replacing it in the template will lead to have @LOCKSPACE_VOLUME_UUID@ as value
2) setting it to empty string will lead to still have the key in the file, just set to an empty string.

let me know what you prefer, but I guess that a value validation (valid uuid) should be done on ha side as well.

Comment 6 Artyom 2014-12-07 07:23:14 UTC
Jiri I opened new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1171452
You can find hosted-engine.conf in attached archive.

Comment 7 Sandro Bonazzola 2015-01-15 14:36:06 UTC
This is an automated message: 
This bug should be fixed in oVirt 3.5.1 RC1, moving to QA

Comment 8 Artyom 2015-01-15 15:16:46 UTC
Verified on ovirt-hosted-engine-ha-1.2.4-5.el6ev.noarch

Comment 9 Sandro Bonazzola 2015-01-21 16:04:07 UTC
oVirt 3.5.1 has been released. If problems still persist, please make note of it in this bug report.