Bug 1163385
| Summary: | Error trying to add new hosted-engine host to upgraded Hosted Engine cluster | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Retired] oVirt | Reporter: | David King <david> | ||||
| Component: | ovirt-hosted-engine-setup | Assignee: | Sandro Bonazzola <sbonazzo> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Artyom <alukiano> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 3.5 | CC: | alukiano, daniel.helgenberger, david, didi, ecohen, gklein, iheim, jmoskovc, lsurette, lveyde, msivak, rbalakri, sbonazzo, stirabos, yeylon | ||||
| Target Milestone: | --- | Keywords: | Triaged | ||||
| Target Release: | 3.5.1 | Flags: | sbonazzo:
ovirt_requires_release_note-
|
||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | integration | ||||||
| Fixed In Version: | ovirt-3.5.1_rc1 | Doc Type: | Bug Fix | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 1163649 (view as bug list) | Environment: | |||||
| Last Closed: | 2015-01-21 16:04:07 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1155170, 1163649, 1164308, 1164311, 1192937 | ||||||
| Attachments: |
|
||||||
In case of updating from 3.4 we need to use FileSystemBackend to handle the storage init while adding new host. idea: if the answer file lacks lockspace uuid and the storage tyupe is nfs we can assume it's 3.4 answer file, so we should use FileSystemBackend I see that fix already in package ovirt-hosted-engine-setup-1.2.1-7.el6ev.noarch,
and setup stage of adding hosts run ok, but after this broker failed to connect to storage:
Thread-481::ERROR::2014-12-04 18:43:59,291::listener::192::ovirt_hosted_engine_ha.broker.listener.ConnectionHandler::(handle) Error handling request, data: 'set-storage-domain VdsmBackend hosted-engine.lockspace=7B22696D6167655F75756964223A20224E6F6E65222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A20224E6F6E65227D sp_uuid=3d30c5d1-a76c-4c79-a6ee-5d92a638e623 dom_type=nfs3 hosted-engine.metadata=7B22696D6167655F75756964223A20224E6F6E65222C202270617468223A206E756C6C2C2022766F6C756D655F75756964223A20224E6F6E65227D sd_uuid=d67ca20f-a4be-4b3e-9a96-78474491111f'
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle
data)
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch
.set_storage_domain(client, sd_type, **options)
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 65, in set_storage_domain
self._backends[client].connect()
File "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 370, in connect
raise RuntimeError(response["status"]["message"])
RuntimeError: Volume does not exist: ('None',)
I looks like because path, broker now receive some incorrect storage details.
So what you prefer, to open new bug or just continue work with this one.
Anyway I will add answer file, broker, vdsm and agent logs.
Created attachment 964755 [details]
logs and answer file
Artyom, can you please also provide hosted-engine.conf. From the code it looks like setup writes the 'None' as the volume_uuid and agent tries to use it as a real value. Sandro, can you make setup to not write that value to config if it's None? (In reply to Artyom from comment #2) > So what you prefer, to open new bug or just continue work with this one. > > Anyway I will add answer file, broker, vdsm and agent logs. I prefer a new BZ. (In reply to Jiri Moskovcak from comment #4) > Sandro, can you make setup to not write that value to config if it's None? 1) not replacing it in the template will lead to have @LOCKSPACE_VOLUME_UUID@ as value 2) setting it to empty string will lead to still have the key in the file, just set to an empty string. let me know what you prefer, but I guess that a value validation (valid uuid) should be done on ha side as well. Jiri I opened new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1171452 You can find hosted-engine.conf in attached archive. This is an automated message: This bug should be fixed in oVirt 3.5.1 RC1, moving to QA Verified on ovirt-hosted-engine-ha-1.2.4-5.el6ev.noarch oVirt 3.5.1 has been released. If problems still persist, please make note of it in this bug report. |
Description of problem: I have upgraded my oVirt 3.4 hosted engine cluster to oVirt 3.5 using the upgrade instructions on the Wiki. Everything appears to be working fine after the upgrade. However, I am now trying to add a new host to the hosted engine configuration but the hosted-engine --deploy fails after sshing the answers file from the upgraded primary configuration. The following errors can be found in the setup log: Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host I confirmed that the answers file on the upgraded host does not have any lockspace UUIDs: OVEHOSTED_STORAGE/storageDatacenterName=str:hosted_datacenter OVEHOSTED_STORAGE/storageDomainName=str:hosted_storage OVEHOSTED_STORAGE/storageType=none:None OVEHOSTED_STORAGE/volUUID=str:da160775-07fe-4569-b45f-03be0c5896a5 OVEHOSTED_STORAGE/domainType=str:nfs3 OVEHOSTED_STORAGE/imgSizeGB=str:25 OVEHOSTED_STORAGE/storageDomainConnection=str:192.168.8.12:/mnt/data2/vm/engine OVEHOSTED_STORAGE/connectionUUID=str:880093ea-b0c1-448d-ac55-cde99feebc23 OVEHOSTED_STORAGE/spUUID=str:5e7ff7c2-6e75-4ba8-a5cc-e8dc5d37e478 OVEHOSTED_STORAGE/imgUUID=str:c9466bb6-a78c-4caa-bce3-22c87a5f3f1a OVEHOSTED_STORAGE/sdUUID=str:b12fd59c-380a-40b3-b7f2-02d455de1d3b Is there something I can do to update the answers file on the updated 3.5 working host so this will work? As an aside the new host is EL7 whereas the existing hosts are EL6. Version-Release number of selected component (if applicable): ovirt-hosted-engine-setup-1.2.1-1.el7.noarch How reproducible: Steps to Reproduce: 1. Install an oVirt 3.4 hosted-engine environment. 2. Upgrade environment to 3.5 using information in Wiki. 3. Attempt to add new hosted-engine host using hosted-engine --deploy Actual results: Adding new host fails with error: Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host Expected results: Able to add new host to hosted-engine cluster. Additional info: Here is the relevant section of the hosted-engine setup log file: 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage validation METHOD otopi.plugins.ovirt_hosted_engine_setup.sanlock.lockspace.Plugin._validation 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/sanlock/lockspace.py", line 102, in _validation 'Answer file lacks lockspace UUIDs, please use an ' RuntimeError: Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host 2014-11-11 22:57:04 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host 2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN 2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True' 2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.RuntimeError'>, RuntimeError('Answer file lacks lockspace UUIDs, please use an answer file generated from the same version you are using on this additional host',), <traceback object at 0x34c85a8>)]' 2014-11-11 22:57:04 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END 2014-11-11 22:57:04 INFO otopi.context context.runSequence:417 Stage: Clean up 2014-11-11 22:57:04 DEBUG otopi.context context.runSequence:421 STAGE cleanup 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.engine.add_host.Plugin._cleanup 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._cleanup 2014-11-11 22:57:04 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._cleanup 2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:609 spmStop 2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._cleanup:970 Not SPM? Traceback (most recent call last): File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 968, in _cleanup self._spmStop() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 617, in _spmStop raise RuntimeError(status_uuid) RuntimeError: Not SPM 2014-11-11 22:57:04 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:580 disconnectStoragePool 2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._cleanup 2014-11-11 22:57:08 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin._save_answers_at_cleanup 2014-11-11 22:57:08 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers:48 Generating answer file '/etc/ovirt-hosted-engine/answers.conf' 2014-11-11 22:57:09 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers_at_cleanup:104 Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated