Bug 1399477 - cannot move master storage domain into maintenance and hosted_storage goes to unknown state
Summary: cannot move master storage domain into maintenance and hosted_storage goes to...
Keywords:
Status: CLOSED DUPLICATE of bug 1405772
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ovirt-4.1.0-beta
: ---
Assignee: Liron Aravot
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-29 06:42 UTC by RamaKasturi
Modified: 2017-01-08 13:55 UTC (History)
2 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2017-01-08 13:55:03 UTC
oVirt Team: Storage
Embargoed:
rule-engine: ovirt-4.1+

Attachments (Terms of Use)

Description RamaKasturi 2016-11-29 06:42:32 UTC 
Description of problem:
I have a scenario where i would want to enable SSL on my existing setup. For enabling SSL i would need to unmount my gluster volumes which are added as storage domains in ovirt. For unmouting the gluster volumes i would need to move the storage domains into maintenance state. I have three domains hosted_storage, vmstore and data (master) . I am able to move vmstore to maintenance. But cannot move master domain (data) to maintenance state and it is stuck in preparing for maintenance and hosted_storage state shows as "unknown".

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.0-0.0.master.20161024211322.gitfc0de31.el7.centos.noarch

How reproducible:
Always

Steps to Reproduce:
1. Have three gluster volumes as storage domains in ovirt.
2. Move domain which is not master into maintenance
3. Now try moving the domain into maintenance which is master.

Actual results:
Cannot move master domain to maintenance and it is stuck in preparing for maintenance state plus hosted_storage goes to unknown state.

Expected results:
Should be able to move master storage domain into maintenance.

Additional info:
Comment 1 RamaKasturi 2016-11-29 06:48:00 UTC 
Following errors are seen in the engine logs:
================================================
2016-11-29 03:07:07,635 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine1 command failed: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:07,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=324, message=Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311']]'
2016-11-29 03:07:07,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [] HostName = hosted_engine1
2016-11-29 03:07:07,635 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [] Command 'ConnectStoragePoolVDSCommand(HostName = hosted_engine1, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e', vdsId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:07,635 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler5) [] FINISH, ConnectStoragePoolVDSCommand, log id: 61c97aa3
2016-11-29 03:07:07,635 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler5) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:07,697 WARN  [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-23) [79d78220] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance
2016-11-29 03:07:07,702 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-28) [] Finished reconstruct for pool '00000001-0001-0001-0001-000000000311'. Clearing event queue
2016-11-29 03:07:08,199 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler1) [] START, GlusterServersListVDSCommand(HostName = hosted_engine1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 42263da4
2016-11-29 03:07:09,341 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler1) [] FINISH, GlusterServersListVDSCommand, return: [10.70.36.79/23:CONNECTED, 10.70.36.80:CONNECTED, 10.70.36.81:CONNECTED], log id: 42263da4
2016-11-29 03:07:09,350 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler1) [] START, GlusterVolumesListVDSCommand(HostName = hosted_engine1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 1b00fda9
2016-11-29 03:07:10,466 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler1) [] FINISH, GlusterVolumesListVDSCommand, return: {8f1770b2-fcfa-4b56-80da-e72a3548ece2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@137866fc, 09d43f7c-a6a2-4f4d-b781-c36e53a48bca=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@86453fd4, 8f9d04bc-1c69-450c-b568-3ab0a1e5e6a6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@84c8a953}, log id: 1b00fda9
2016-11-29 03:07:15,575 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] START, GlusterServersListVDSCommand(HostName = hosted_engine1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: aa97c1d
2016-11-29 03:07:16,635 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] FINISH, GlusterServersListVDSCommand, return: [10.70.36.79/23:CONNECTED, 10.70.36.80:CONNECTED, 10.70.36.81:CONNECTED], log id: aa97c1d
2016-11-29 03:07:16,644 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] START, GlusterVolumesListVDSCommand(HostName = hosted_engine1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 3a7f719a
2016-11-29 03:07:17,768 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] START, ConnectStoragePoolVDSCommand(HostName = hosted_engine3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962', vdsId='728ee3c2-5292-4fac-9ad4-ee9266413962', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'}), log id: 5bfe0c88
2016-11-29 03:07:17,769 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] Executing with domain map: {27f054c3-c245-4039-b42a-c28b37043016=active, f3b0e738-03e9-49a1-886c-aa021cd8badb=attached, 5ba64954-f23a-4c48-b62c-06db29bf4891=attached}
2016-11-29 03:07:17,776 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] FINISH, GlusterVolumesListVDSCommand, return: {8f1770b2-fcfa-4b56-80da-e72a3548ece2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@137866fc, 09d43f7c-a6a2-4f4d-b781-c36e53a48bca=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@86453fd4, 8f9d04bc-1c69-450c-b568-3ab0a1e5e6a6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@84c8a953}, log id: 3a7f719a
2016-11-29 03:07:18,897 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler7) [464f038e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine3 command failed: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:18,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=324, message=Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311']]'
2016-11-29 03:07:18,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] HostName = hosted_engine3
2016-11-29 03:07:18,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] Command 'ConnectStoragePoolVDSCommand(HostName = hosted_engine3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962', vdsId='728ee3c2-5292-4fac-9ad4-ee9266413962', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:18,897 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler7) [464f038e] FINISH, ConnectStoragePoolVDSCommand, log id: 5bfe0c88
2016-11-29 03:07:18,897 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler7) [464f038e] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:18,920 WARN  [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-33) [335dc01c] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance
2016-11-29 03:07:18,926 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-34) [464f038e] Finished reconstruct for pool '00000001-0001-0001-0001-000000000311'. Clearing event queue
2016-11-29 03:07:22,854 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [] START, GlusterServersListVDSCommand(HostName = hosted_engine1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: b6e6ee2
2016-11-29 03:07:23,211 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler2) [] FINISH, GlusterServersListVDSCommand, return: [10.70.36.79/23:CONNECTED, 10.70.36.80:CONNECTED, 10.70.36.81:CONNECTED], log id: b6e6ee2
2016-11-29 03:07:23,217 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [] START, GlusterVolumesListVDSCommand(HostName = hosted_engine1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 17892b0c
2016-11-29 03:07:24,320 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler2) [] FINISH, GlusterVolumesListVDSCommand, return: {8f1770b2-fcfa-4b56-80da-e72a3548ece2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@137866fc, 09d43f7c-a6a2-4f4d-b781-c36e53a48bca=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@86453fd4, 8f9d04bc-1c69-450c-b568-3ab0a1e5e6a6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@84c8a953}, log id: 17892b0c
2016-11-29 03:07:28,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] START, ConnectStoragePoolVDSCommand(HostName = hosted_engine3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962', vdsId='728ee3c2-5292-4fac-9ad4-ee9266413962', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'}), log id: 2c675659
2016-11-29 03:07:28,968 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] Executing with domain map: {27f054c3-c245-4039-b42a-c28b37043016=active, f3b0e738-03e9-49a1-886c-aa021cd8badb=attached, 5ba64954-f23a-4c48-b62c-06db29bf4891=attached}
2016-11-29 03:07:29,400 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [] START, GlusterServersListVDSCommand(HostNam:e = hosted_engine1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 1c41326b
2016-11-29 03:07:29,405 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler1) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine3 command failed: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:29,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=324, message=Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311']]'
2016-11-29 03:07:29,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] HostName = hosted_engine3
2016-11-29 03:07:29,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] Command 'ConnectStoragePoolVDSCommand(HostName = hosted_engine3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962', vdsId='728ee3c2-5292-4fac-9ad4-ee9266413962', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:29,405 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler1) [] FINISH, ConnectStoragePoolVDSCommand, log id: 2c675659
2016-11-29 03:07:29,405 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler1) [] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:29,424 WARN  [org.ovirt.engine.core.bll.storage.pool.ReconstructMasterDomainCommand] (org.ovirt.thread.pool-6-thread-42) [b371c3e] Validation of action 'ReconstructMasterDomain' failed for user SYSTEM. Reasons: VAR__ACTION__RECONSTRUCT_MASTER,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL2,$status PreparingForMaintenance
:
2016-11-29 03:07:29,429 INFO  [org.ovirt.engine.core.bll.eventqueue.EventQueueMonitor] (org.ovirt.thread.pool-6-thread-39) [] Finished reconstruct for pool '00000001-0001-0001-0001-000000000311'. Clearing event queue
2016-11-29 03:07:30,544 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [] FINISH, GlusterServersListVDSCommand, return: [10.70.36.79/23:CONNECTED, 10.70.36.80:CONNECTED, 10.70.36.81:CONNECTED], log id: 1c41326b
2016-11-29 03:07:30,552 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [] START, GlusterVolumesListVDSCommand(HostName = hosted_engine1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 4a5b61bb
2016-11-29 03:07:30,678 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [] FINISH, GlusterVolumesListVDSCommand, return: {8f1770b2-fcfa-4b56-80da-e72a3548ece2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@137866fc, 09d43f7c-a6a2-4f4d-b781-c36e53a48bca=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@86453fd4, 8f9d04bc-1c69-450c-b568-3ab0a1e5e6a6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@84c8a953}, log id: 4a5b61bb
2016-11-29 03:07:34,048 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [] START, GlusterTasksListVDSCommand(HostName = hosted_engine3, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962'}), log id: 518894bd
2016-11-29 03:07:35,090 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [] FINISH, GlusterTasksListVDSCommand, return: [], log id: 518894bd
2016-11-29 03:07:35,757 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] START, GlusterServersListVDSCommand(HostName = hosted_engine1, VdsIdVDSCommandParametersBase:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 49b9680a
2016-11-29 03:07:36,917 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] FINISH, GlusterServersListVDSCommand, return: [10.70.36.79/23:CONNECTED, 10.70.36.80:CONNECTED, 10.70.36.81:CONNECTED], log id: 49b9680a
2016-11-29 03:07:36,926 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] START, GlusterVolumesListVDSCommand(HostName = hosted_engine1, GlusterVolumesListVDSParameters:{runAsync='true', hostId='0695e6b0-e6f3-4c1d-b29d-b44bdbe8fa0e'}), log id: 499abdff
2016-11-29 03:07:38,029 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler10) [5ffd5b70] FINISH, GlusterVolumesListVDSCommand, return: {8f1770b2-fcfa-4b56-80da-e72a3548ece2=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@137866fc, 09d43f7c-a6a2-4f4d-b781-c36e53a48bca=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@86453fd4, 8f9d04bc-1c69-450c-b568-3ab0a1e5e6a6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@84c8a953}, log id: 499abdff
2016-11-29 03:07:39,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [] START, ConnectStoragePoolVDSCommand(HostName = hosted_engine3, ConnectStoragePoolVDSCommandParameters:{runAsync='true', hostId='728ee3c2-5292-4fac-9ad4-ee9266413962', vdsId='728ee3c2-5292-4fac-9ad4-ee9266413962', storagePoolId='00000001-0001-0001-0001-000000000311', masterVersion='1'}), log id: 27c7c2ae
2016-11-29 03:07:39,476 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [] Executing with domain map: {27f054c3-c245-4039-b42a-c28b37043016=active, f3b0e738-03e9-49a1-886c-aa021cd8badb=attached, 5ba64954-f23a-4c48-b62c-06db29bf4891=attached}
2016-11-29 03:07:40,602 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler3) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM hosted_engine3 command failed: Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311'
2016-11-29 03:07:40,602 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (DefaultQuartzScheduler3) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand' return value 'StatusOnlyReturnForXmlRpc [status=StatusForXmlRpc [code=324, message=Wrong Master domain or its version: u'SD=f3b0e738-03e9-49a1-886c-aa021cd8badb, pool=00000001-0001-0001-0001-000000000311']]'
Comment 2 RamaKasturi 2016-11-29 07:21:27 UTC 
sosreports can be found in the link below:
===================================================
http://rhsqe-repo.lab.eng.blr.redhat.com/sosreports/HC/1399477/
Comment 3 Liron Aravot 2017-01-08 13:55:03 UTC 

*** This bug has been marked as a duplicate of bug 1405772 ***
Note You need to log in before you can comment on or make changes to this bug.