Bug 1291234

Summary: with 3.6.1, on additional hosts, the deploy fails with 'RuntimeError: Dirty Storage Domain: Cannot find master domain...'
Product: [oVirt] ovirt-hosted-engine-setup Reporter: Simone Tiraboschi <stirabos>
Component: Plugins.GeneralAssignee: Simone Tiraboschi <stirabos>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.3.1CC: bugs, cshao, didi, duckd, fdeutsch, huiwa, lveyde, mavital, rmartins, sbonazzo, stirabos, ycui, ylavi
Target Milestone: ovirt-3.6.1Keywords: Triaged
Target Release: 1.3.1.3Flags: rule-engine: ovirt-3.6.z+
rule-engine: blocker+
ylavi: planning_ack+
sbonazzo: devel_ack+
mavital: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: ovirt-hosted-engine-setup-1.3.1.3-1 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-12-16 12:20:17 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1291736    
Bug Blocks: 1284979, 1285700    

Description Simone Tiraboschi 2015-12-14 11:35:51 UTC
Description of problem:

Before 3.6.1 _storagePoolConnection was not required on additional hosts since the hosted-engine storage domain wasn't connected to any storage pool.

With 3.6.1 the engine is able to import the hosted-engine storage domain and attach it to its datacenter storagePool.
So _storagePoolConnection is needed again on additional hosts and we have to fix it since the master storage domain is no the bootstrap fake storage domain but the hosted-engine-storage domain.

Version-Release number of selected component (if applicable):
ovirt-hosted-engine-setup.noarch      1.3.1.2-1.el7.centos        @ovirt-3.6-pre

How reproducible:
100%

Steps to Reproduce:
1. deploy hosted-engine on first host, use at least 3.6.1 RC4 
2. ensure that the hosted-engine storage domain got imported and activated within the engine 
3. try to deploy an additional host

Actual results:
It's not possible to deploy additional hosts, the setup fails with:
2015-12-14 11:40:02 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._handleHostId:249 Installing on additional host
2015-12-14 11:40:02 DEBUG otopi.plugins.otopi.dialog.human human.queryString:156 query OVEHOSTED_HOST_ID
2015-12-14 11:40:02 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:SEND                 Please specify the Host ID [Must be integer, default: 2]: 
2015-12-14 11:40:05 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:219 DIALOG:RECEIVE    3
2015-12-14 11:40:05 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:922 connectStoragePool
2015-12-14 11:40:05 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 1265, in _late_customization
    self._getExistingDomain()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 559, in _getExistingDomain
    self._storagePoolConnection()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 930, in _storagePoolConnection
    message=status['status']['message'],
RuntimeError: Dirty Storage Domain: Cannot find master domain: 'spUUID=00000001-0001-0001-0001-00000000009e, msdUUID=154ef8b4-fad7-44cb-b7e7-25af78a51f98'
Please clean the storage device and try again
2015-12-14 11:40:05 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Environment customization': Dirty Storage Domain: Cannot find master domain: 'spUUID=00000001-0001-0001-0001-00000000009e, msdUUID=154ef8b4-fad7-44cb-b7e7-25af78a51f98'
Please clean the storage device and try again


Expected results:
it correctly deploy the additional host.

Additional info:
The hosted-engine storage domain must be visible and active in the engine to trigger it.

Comment 1 Nikolai Sednev 2015-12-15 13:34:49 UTC
Successfully added third host to the HE with 2 hosts running, met no errors as described within the bug.
Works for me on these components:
Host:
qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
vdsm-4.17.13-1.el7ev.noarch
sanlock-3.2.4-2.el7_2.x86_64
ovirt-setup-lib-1.0.0-1.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch
mom-0.5.1-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch
Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015

Engine:
ovirt-host-deploy-java-1.4.1-1.el6ev.noarch
ovirt-vmconsole-1.0.0-1.el6ev.noarch
ovirt-host-deploy-1.4.1-1.el6ev.noarch
ovirt-vmconsole-proxy-1.0.0-1.el6ev.noarch
ovirt-engine-extension-aaa-jdbc-1.0.4-1.el6ev.noarch
rhevm-3.6.1.3-0.1.el6.noarch
Linux version 2.6.32-573.8.1.el6.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Fri Sep 25 19:24:22 EDT 2015

Comment 2 Sandro Bonazzola 2015-12-16 12:20:17 UTC
According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.

Comment 3 Douglas Duckworth 2016-03-07 20:22:55 UTC
Workaround:

In RHEV-M 3.5.7-0.1.el6ev on additional RHEV Hypervisor - 7.2 - 20160219.0.el7ev I was able to add additional Hosted-Engine Host by doing:

hosted-engine --set-maintenance --mode=global
hosted-engine  --vm-stop
hosted-engine  --deploy

Dirty Storage Domain message gone.